2023-08-26T13:14:44.325 INFO:root:teuthology version: 0.0.1.dev185+g449a1bc 2023-08-26T13:14:44.326 DEBUG:teuthology.run:Teuthology command: teuthology --verbose --description rbd/mirror-thrash/{base/install clients/mirror cluster/{2-node openstack} msgr-failures/few objectstore/bluestore-stupid policy/simple rbd-mirror/four-per-cluster supported-random-distro$/{rhel_8} workloads/rbd-mirror-snapshot-stress-workunit-exclusive-lock} --name yuriw-2023-08-25_17:40:36-rbd-main-distro-default-smithi --owner scheduled_yuriw@teuthology --archive /home/teuthworker/archive/yuriw-2023-08-25_17:40:36-rbd-main-distro-default-smithi/7380431 -- /home/teuthworker/archive/yuriw-2023-08-25_17:40:36-rbd-main-distro-default-smithi/7380431/orig.config.yaml 2023-08-26T13:14:44.384 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2023-08-26T13:14:44.473 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/yuriw-2023-08-25_17:40:36-rbd-main-distro-default-smithi/7380431 branch: main description: rbd/mirror-thrash/{base/install clients/mirror cluster/{2-node openstack} msgr-failures/few objectstore/bluestore-stupid policy/simple rbd-mirror/four-per-cluster supported-random-distro$/{rhel_8} workloads/rbd-mirror-snapshot-stress-workunit-exclusive-lock} email: yweinste@redhat.com first_in_suite: false job_id: '7380431' kernel: kdb: true sha1: distro ktype: distro last_in_suite: false machine_type: smithi meta: - desc: run two ceph clusters and install rbd-mirror - desc: configure the permissions for client.mirror - desc: 2 ceph clusters with 1 mon and 3 osds each - desc: run four rbd-mirror daemons per cluster - desc: run the rbd_mirror_stress.sh workunit to test the rbd-mirror daemon name: yuriw-2023-08-25_17:40:36-rbd-main-distro-default-smithi no_nested_subset: false nuke-on-error: true openstack: - volumes: count: 3 size: 30 os_type: rhel os_version: '8.6' overrides: admin_socket: branch: main ceph: conf: client: debug rbd: 20 debug rbd_mirror: 15 log to stderr: false rbd default features: 125 rbd mirror image policy type: simple client.mirror.0: admin socket: /var/run/ceph/rbd-mirror.$cluster-$name.asok pid file: /var/run/ceph/rbd-mirror.$cluster-$name.pid client.mirror.1: admin socket: /var/run/ceph/rbd-mirror.$cluster-$name.asok pid file: /var/run/ceph/rbd-mirror.$cluster-$name.pid client.mirror.2: admin socket: /var/run/ceph/rbd-mirror.$cluster-$name.asok pid file: /var/run/ceph/rbd-mirror.$cluster-$name.pid client.mirror.3: admin socket: /var/run/ceph/rbd-mirror.$cluster-$name.asok pid file: /var/run/ceph/rbd-mirror.$cluster-$name.pid global: mon client directed command retry: 5 ms inject socket failures: 5000 mgr: debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 osd: bdev async discard: true bdev enable discard: true bluestore allocator: stupid bluestore block size: 96636764160 bluestore fsck on mount: true debug bluefs: 1/20 debug bluestore: 1/20 debug ms: 1 debug osd: 20 debug rocksdb: 4/10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd failsafe full ratio: 0.95 osd objectstore: bluestore flavor: default fs: xfs log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - \(OSD_SLOW_PING_TIME sha1: e7eb9853fddaf661c68f207346131a7e00d8ed2f ceph-deploy: bluestore: true conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: {} osd: bdev async discard: true bdev enable discard: true bluestore block size: 96636764160 bluestore fsck on mount: true debug bluefs: 1/20 debug bluestore: 1/20 debug rocksdb: 4/10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd failsafe full ratio: 0.95 osd objectstore: bluestore fs: xfs install: ceph: flavor: default sha1: e7eb9853fddaf661c68f207346131a7e00d8ed2f selinux: whitelist: - scontext=system_u:system_r:logrotate_t:s0 thrashosds: bdev_inject_crash: 2 bdev_inject_crash_probability: 0.5 workunit: branch: main sha1: e7eb9853fddaf661c68f207346131a7e00d8ed2f owner: scheduled_yuriw@teuthology priority: 75 repo: https://github.com/ceph/ceph-ci.git roles: - - cluster1.mon.a - cluster1.mgr.x - cluster2.mgr.x - cluster1.osd.0 - cluster1.osd.1 - cluster1.osd.2 - cluster1.client.0 - cluster2.client.0 - - cluster2.mon.a - cluster2.osd.0 - cluster2.osd.1 - cluster2.osd.2 - cluster1.client.mirror - cluster1.client.mirror.0 - cluster1.client.mirror.1 - cluster1.client.mirror.2 - cluster1.client.mirror.3 - cluster1.client.mirror.4 - cluster1.client.mirror.5 - cluster1.client.mirror.6 - cluster2.client.mirror - cluster2.client.mirror.0 - cluster2.client.mirror.1 - cluster2.client.mirror.2 - cluster2.client.mirror.3 - cluster2.client.mirror.4 - cluster2.client.mirror.5 - cluster2.client.mirror.6 seed: 5715 sha1: e7eb9853fddaf661c68f207346131a7e00d8ed2f sleep_before_teardown: 0 subset: 111/120000 suite: rbd suite_branch: main suite_path: /home/teuthworker/src/github.com_ceph_ceph-c_e7eb9853fddaf661c68f207346131a7e00d8ed2f/qa suite_relpath: qa suite_repo: https://github.com/ceph/ceph-ci.git suite_sha1: e7eb9853fddaf661c68f207346131a7e00d8ed2f targets: smithi031.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCpUxo5ay/ucdkX9rleNMm8Ofp24XRXUtJPOK3Ji3XBpTiUgmmwneaY6316+3CG+k4K0p3IebanuyTvZApjm02/dNCvu8yD3rjEpWVgIuqYH8O3lEdN9DXcCX+7xHJE0m18+ZPsJQVsNrNJxY+5zqyNnBc8jpZq2LStsWwPOKPWhauANRkOBFSGDoLppWZbLx/yJs+R0qoNLLK9LpKQTUnqsapXvbD27OpPahzj/qbNxFPPyY6XrrldZ6qEO+EjJOujFLH3gzv0kJu10DCuzxrpz6ston4RPi1O3GgcHnF4Mqr/W9VwZ94UTMh+k4dIFRVqJEKrcwRHa5kFsvH/WhVx0Ygr/MiV/Q86mmV7JT+/MpzIRTtWyei5L9IBeZIE9cwzj+mOk6SD/t1roIVEGVV/uF2AyFDVBPr6F8oZ822OGpUyyaDsWTitPNdTX4TC3zdSrLc9tyDa1P74eH/ZLeTSf+C6pM/cepOhQaa6XzpDiPSuJUm9HOQxUlEAvcqa7Ac= smithi124.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCpUxo5ay/ucdkX9rleNMm8Ofp24XRXUtJPOK3Ji3XBpTiUgmmwneaY6316+3CG+k4K0p3IebanuyTvZApjm02/dNCvu8yD3rjEpWVgIuqYH8O3lEdN9DXcCX+7xHJE0m18+ZPsJQVsNrNJxY+5zqyNnBc8jpZq2LStsWwPOKPWhauANRkOBFSGDoLppWZbLx/yJs+R0qoNLLK9LpKQTUnqsapXvbD27OpPahzj/qbNxFPPyY6XrrldZ6qEO+EjJOujFLH3gzv0kJu10DCuzxrpz6ston4RPi1O3GgcHnF4Mqr/W9VwZ94UTMh+k4dIFRVqJEKrcwRHa5kFsvH/WhVx0Ygr/MiV/Q86mmV7JT+/MpzIRTtWyei5L9IBeZIE9cwzj+mOk6SD/t1roIVEGVV/uF2AyFDVBPr6F8oZ822OGpUyyaDsWTitPNdTX4TC3zdSrLc9tyDa1P74eH/ZLeTSf+C6pM/cepOhQaa6XzpDiPSuJUm9HOQxUlEAvcqa7Ac= tasks: - install: extra_packages: - rbd-mirror - ceph: cluster: cluster1 - ceph: cluster: cluster2 - exec: cluster1.client.mirror.0: - sudo ceph --cluster cluster1 auth caps client.mirror mon 'profile rbd-mirror-peer' osd 'profile rbd' - sudo ceph --cluster cluster1 auth caps client.mirror.0 mon 'profile rbd-mirror' osd 'profile rbd' - sudo ceph --cluster cluster1 auth caps client.mirror.1 mon 'profile rbd-mirror' osd 'profile rbd' - sudo ceph --cluster cluster1 auth caps client.mirror.2 mon 'profile rbd-mirror' osd 'profile rbd' - sudo ceph --cluster cluster1 auth caps client.mirror.3 mon 'profile rbd-mirror' osd 'profile rbd' - sudo ceph --cluster cluster2 auth caps client.mirror mon 'profile rbd-mirror-peer' osd 'profile rbd' - sudo ceph --cluster cluster2 auth caps client.mirror.0 mon 'profile rbd-mirror' osd 'profile rbd' - sudo ceph --cluster cluster2 auth caps client.mirror.1 mon 'profile rbd-mirror' osd 'profile rbd' - sudo ceph --cluster cluster2 auth caps client.mirror.2 mon 'profile rbd-mirror' osd 'profile rbd' - sudo ceph --cluster cluster2 auth caps client.mirror.3 mon 'profile rbd-mirror' osd 'profile rbd' - rbd-mirror: client: cluster1.client.mirror.0 thrash: true - rbd-mirror: client: cluster1.client.mirror.1 thrash: true - rbd-mirror: client: cluster1.client.mirror.2 thrash: true - rbd-mirror: client: cluster1.client.mirror.3 thrash: true - rbd-mirror: client: cluster2.client.mirror.0 thrash: true - rbd-mirror: client: cluster2.client.mirror.1 thrash: true - rbd-mirror: client: cluster2.client.mirror.2 thrash: true - rbd-mirror: client: cluster2.client.mirror.3 thrash: true - rbd-mirror-thrash: cluster: cluster1 - rbd-mirror-thrash: cluster: cluster2 - workunit: clients: cluster1.client.mirror: - rbd/rbd_mirror_stress.sh env: CEPH_ARGS: '' MIRROR_IMAGE_MODE: snapshot MIRROR_POOL_MODE: image RBD_IMAGE_FEATURES: layering,exclusive-lock RBD_MIRROR_INSTANCES: '4' RBD_MIRROR_USE_EXISTING_CLUSTER: '1' RBD_MIRROR_USE_RBD_MIRROR: '1' timeout: 6h teuthology: fragments_dropped: [] meta: {} postmerge: [] teuthology_branch: main teuthology_sha1: 449a1bc2027504e7b3c3d7b30fa4178906581da7 timestamp: 2023-08-25_17:40:36 tube: smithi user: yuriw verbose: true worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.2002 2023-08-26T13:14:44.473 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/github.com_ceph_ceph-c_e7eb9853fddaf661c68f207346131a7e00d8ed2f/qa; will attempt to use it 2023-08-26T13:14:44.474 INFO:teuthology.run:Found tasks at /home/teuthworker/src/github.com_ceph_ceph-c_e7eb9853fddaf661c68f207346131a7e00d8ed2f/qa/tasks 2023-08-26T13:14:44.474 INFO:teuthology.run_tasks:Running task internal.check_packages... 2023-08-26T13:14:44.477 INFO:teuthology.task.internal:Checking packages... 2023-08-26T13:14:44.500 INFO:teuthology.task.internal:Checking packages for os_type 'rhel', flavor 'default' and ceph hash 'e7eb9853fddaf661c68f207346131a7e00d8ed2f' 2023-08-26T13:14:44.500 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2023-08-26T13:14:44.501 INFO:teuthology.packaging:ref: None 2023-08-26T13:14:44.501 INFO:teuthology.packaging:tag: None 2023-08-26T13:14:44.501 INFO:teuthology.packaging:branch: main 2023-08-26T13:14:44.501 INFO:teuthology.packaging:sha1: e7eb9853fddaf661c68f207346131a7e00d8ed2f 2023-08-26T13:14:44.502 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&ref=main 2023-08-26T13:14:44.784 INFO:teuthology.task.internal:Found packages for ceph version 18.0.0-5708.gd50a53f8 2023-08-26T13:14:44.785 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2023-08-26T13:14:44.792 INFO:teuthology.task.internal:no buildpackages task found 2023-08-26T13:14:44.792 INFO:teuthology.run_tasks:Running task internal.save_config... 2023-08-26T13:14:44.801 INFO:teuthology.task.internal:Saving configuration 2023-08-26T13:14:44.820 INFO:teuthology.run_tasks:Running task internal.check_lock... 2023-08-26T13:14:44.826 INFO:teuthology.task.internal.check_lock:Checking locks... 2023-08-26T13:14:44.849 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi031.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yuriw-2023-08-25_17:40:36-rbd-main-distro-default-smithi/7380431', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'rhel', 'os_version': '8.6', 'arch': 'x86_64', 'locked': True, 'locked_since': '2023-08-26 13:09:35.383564', 'locked_by': 'scheduled_yuriw@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBLVvN3+LmzPZ4ixiwyCkS3KZLZ9R6MOPm+kPtaJbl8fExpzbk8TudXCcJWHBL7DGPWFvye2jT6sei9eiX0EUQYs='} 2023-08-26T13:14:44.869 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi124.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yuriw-2023-08-25_17:40:36-rbd-main-distro-default-smithi/7380431', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'rhel', 'os_version': '8.6', 'arch': 'x86_64', 'locked': True, 'locked_since': '2023-08-26 13:09:35.381920', 'locked_by': 'scheduled_yuriw@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBLVvN3+LmzPZ4ixiwyCkS3KZLZ9R6MOPm+kPtaJbl8fExpzbk8TudXCcJWHBL7DGPWFvye2jT6sei9eiX0EUQYs='} 2023-08-26T13:14:44.869 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2023-08-26T13:14:44.877 INFO:teuthology.task.internal:roles: ubuntu@smithi031.front.sepia.ceph.com - ['cluster1.mon.a', 'cluster1.mgr.x', 'cluster2.mgr.x', 'cluster1.osd.0', 'cluster1.osd.1', 'cluster1.osd.2', 'cluster1.client.0', 'cluster2.client.0'] 2023-08-26T13:14:44.878 INFO:teuthology.task.internal:roles: ubuntu@smithi124.front.sepia.ceph.com - ['cluster2.mon.a', 'cluster2.osd.0', 'cluster2.osd.1', 'cluster2.osd.2', 'cluster1.client.mirror', 'cluster1.client.mirror.0', 'cluster1.client.mirror.1', 'cluster1.client.mirror.2', 'cluster1.client.mirror.3', 'cluster1.client.mirror.4', 'cluster1.client.mirror.5', 'cluster1.client.mirror.6', 'cluster2.client.mirror', 'cluster2.client.mirror.0', 'cluster2.client.mirror.1', 'cluster2.client.mirror.2', 'cluster2.client.mirror.3', 'cluster2.client.mirror.4', 'cluster2.client.mirror.5', 'cluster2.client.mirror.6'] 2023-08-26T13:14:44.878 INFO:teuthology.run_tasks:Running task console_log... 2023-08-26T13:14:44.963 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7f8f98751f70>, signals=[15]) 2023-08-26T13:14:44.965 INFO:teuthology.run_tasks:Running task internal.connect... 2023-08-26T13:14:44.980 INFO:teuthology.task.internal:Opening connections... 2023-08-26T13:14:44.980 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi031.front.sepia.ceph.com 2023-08-26T13:14:44.982 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi031.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-08-26T13:14:45.054 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi124.front.sepia.ceph.com 2023-08-26T13:14:45.055 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi124.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-08-26T13:14:45.122 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2023-08-26T13:14:45.129 DEBUG:teuthology.orchestra.run.smithi031:> uname -m 2023-08-26T13:14:45.151 INFO:teuthology.orchestra.run.smithi031.stdout:x86_64 2023-08-26T13:14:45.152 DEBUG:teuthology.orchestra.run.smithi031:> cat /etc/os-release 2023-08-26T13:14:45.213 INFO:teuthology.orchestra.run.smithi031.stdout:NAME="Red Hat Enterprise Linux" 2023-08-26T13:14:45.213 INFO:teuthology.orchestra.run.smithi031.stdout:VERSION="8.6 (Ootpa)" 2023-08-26T13:14:45.213 INFO:teuthology.orchestra.run.smithi031.stdout:ID="rhel" 2023-08-26T13:14:45.213 INFO:teuthology.orchestra.run.smithi031.stdout:ID_LIKE="fedora" 2023-08-26T13:14:45.214 INFO:teuthology.orchestra.run.smithi031.stdout:VERSION_ID="8.6" 2023-08-26T13:14:45.214 INFO:teuthology.orchestra.run.smithi031.stdout:PLATFORM_ID="platform:el8" 2023-08-26T13:14:45.214 INFO:teuthology.orchestra.run.smithi031.stdout:PRETTY_NAME="Red Hat Enterprise Linux 8.6 (Ootpa)" 2023-08-26T13:14:45.215 INFO:teuthology.orchestra.run.smithi031.stdout:ANSI_COLOR="0;31" 2023-08-26T13:14:45.215 INFO:teuthology.orchestra.run.smithi031.stdout:CPE_NAME="cpe:/o:redhat:enterprise_linux:8::baseos" 2023-08-26T13:14:45.215 INFO:teuthology.orchestra.run.smithi031.stdout:HOME_URL="https://www.redhat.com/" 2023-08-26T13:14:45.215 INFO:teuthology.orchestra.run.smithi031.stdout:DOCUMENTATION_URL="https://access.redhat.com/documentation/red_hat_enterprise_linux/8/" 2023-08-26T13:14:45.216 INFO:teuthology.orchestra.run.smithi031.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2023-08-26T13:14:45.216 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-08-26T13:14:45.216 INFO:teuthology.orchestra.run.smithi031.stdout:REDHAT_BUGZILLA_PRODUCT="Red Hat Enterprise Linux 8" 2023-08-26T13:14:45.216 INFO:teuthology.orchestra.run.smithi031.stdout:REDHAT_BUGZILLA_PRODUCT_VERSION=8.6 2023-08-26T13:14:45.216 INFO:teuthology.orchestra.run.smithi031.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux" 2023-08-26T13:14:45.217 INFO:teuthology.orchestra.run.smithi031.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="8.6" 2023-08-26T13:14:45.217 INFO:teuthology.lock.ops:Updating smithi031.front.sepia.ceph.com on lock server 2023-08-26T13:14:45.236 DEBUG:teuthology.orchestra.run.smithi124:> uname -m 2023-08-26T13:14:45.256 INFO:teuthology.orchestra.run.smithi124.stdout:x86_64 2023-08-26T13:14:45.257 DEBUG:teuthology.orchestra.run.smithi124:> cat /etc/os-release 2023-08-26T13:14:45.316 INFO:teuthology.orchestra.run.smithi124.stdout:NAME="Red Hat Enterprise Linux" 2023-08-26T13:14:45.316 INFO:teuthology.orchestra.run.smithi124.stdout:VERSION="8.6 (Ootpa)" 2023-08-26T13:14:45.316 INFO:teuthology.orchestra.run.smithi124.stdout:ID="rhel" 2023-08-26T13:14:45.316 INFO:teuthology.orchestra.run.smithi124.stdout:ID_LIKE="fedora" 2023-08-26T13:14:45.317 INFO:teuthology.orchestra.run.smithi124.stdout:VERSION_ID="8.6" 2023-08-26T13:14:45.317 INFO:teuthology.orchestra.run.smithi124.stdout:PLATFORM_ID="platform:el8" 2023-08-26T13:14:45.317 INFO:teuthology.orchestra.run.smithi124.stdout:PRETTY_NAME="Red Hat Enterprise Linux 8.6 (Ootpa)" 2023-08-26T13:14:45.317 INFO:teuthology.orchestra.run.smithi124.stdout:ANSI_COLOR="0;31" 2023-08-26T13:14:45.318 INFO:teuthology.orchestra.run.smithi124.stdout:CPE_NAME="cpe:/o:redhat:enterprise_linux:8::baseos" 2023-08-26T13:14:45.318 INFO:teuthology.orchestra.run.smithi124.stdout:HOME_URL="https://www.redhat.com/" 2023-08-26T13:14:45.318 INFO:teuthology.orchestra.run.smithi124.stdout:DOCUMENTATION_URL="https://access.redhat.com/documentation/red_hat_enterprise_linux/8/" 2023-08-26T13:14:45.318 INFO:teuthology.orchestra.run.smithi124.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2023-08-26T13:14:45.318 INFO:teuthology.orchestra.run.smithi124.stdout: 2023-08-26T13:14:45.319 INFO:teuthology.orchestra.run.smithi124.stdout:REDHAT_BUGZILLA_PRODUCT="Red Hat Enterprise Linux 8" 2023-08-26T13:14:45.319 INFO:teuthology.orchestra.run.smithi124.stdout:REDHAT_BUGZILLA_PRODUCT_VERSION=8.6 2023-08-26T13:14:45.319 INFO:teuthology.orchestra.run.smithi124.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux" 2023-08-26T13:14:45.319 INFO:teuthology.orchestra.run.smithi124.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="8.6" 2023-08-26T13:14:45.320 INFO:teuthology.lock.ops:Updating smithi124.front.sepia.ceph.com on lock server 2023-08-26T13:14:45.344 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2023-08-26T13:14:45.354 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2023-08-26T13:14:45.364 INFO:teuthology.task.internal:Checking for old test directory... 2023-08-26T13:14:45.364 DEBUG:teuthology.orchestra.run.smithi031:> test '!' -e /home/ubuntu/cephtest 2023-08-26T13:14:45.367 DEBUG:teuthology.orchestra.run.smithi124:> test '!' -e /home/ubuntu/cephtest 2023-08-26T13:14:45.385 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2023-08-26T13:14:45.391 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2023-08-26T13:14:45.391 DEBUG:teuthology.orchestra.run.smithi031:> test -z $(ls -A /var/lib/ceph) 2023-08-26T13:14:45.427 DEBUG:teuthology.orchestra.run.smithi124:> test -z $(ls -A /var/lib/ceph) 2023-08-26T13:14:45.456 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2023-08-26T13:14:45.510 INFO:teuthology.run_tasks:Running task kernel... 2023-08-26T13:14:45.527 INFO:teuthology.task.kernel:normalize config orig: {'kdb': True, 'sha1': 'distro'} 2023-08-26T13:14:45.528 INFO:teuthology.task.kernel:config {'cluster1.mon.a': {'kdb': True, 'sha1': 'distro'}, 'cluster2.mon.a': {'kdb': True, 'sha1': 'distro'}}, timeout 300 2023-08-26T13:14:45.528 DEBUG:teuthology.orchestra.run.smithi031:> test -f /run/.containerenv -o -f /.dockerenv 2023-08-26T13:14:45.529 DEBUG:teuthology.orchestra.run.smithi124:> test -f /run/.containerenv -o -f /.dockerenv 2023-08-26T13:14:45.548 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-08-26T13:14:45.548 DEBUG:teuthology.orchestra.run.smithi124:> uname -r 2023-08-26T13:14:45.549 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-08-26T13:14:45.550 DEBUG:teuthology.orchestra.run.smithi031:> uname -r 2023-08-26T13:14:45.607 INFO:teuthology.orchestra.run.smithi124.stdout:4.18.0-372.9.1.el8.x86_64 2023-08-26T13:14:45.608 INFO:teuthology.task.kernel:Running kernel on smithi124: 4.18.0-372.9.1.el8.x86_64 2023-08-26T13:14:45.608 DEBUG:teuthology.orchestra.run.smithi124:> sudo yum install -y kernel 2023-08-26T13:14:45.610 INFO:teuthology.orchestra.run.smithi031.stdout:4.18.0-372.9.1.el8.x86_64 2023-08-26T13:14:45.610 INFO:teuthology.task.kernel:Running kernel on smithi031: 4.18.0-372.9.1.el8.x86_64 2023-08-26T13:14:45.610 DEBUG:teuthology.orchestra.run.smithi031:> sudo yum install -y kernel 2023-08-26T13:14:46.762 INFO:teuthology.orchestra.run.smithi031.stdout:Updating Subscription Management repositories. 2023-08-26T13:14:46.762 INFO:teuthology.orchestra.run.smithi031.stdout:Unable to read consumer identity 2023-08-26T13:14:46.783 INFO:teuthology.orchestra.run.smithi124.stdout:Updating Subscription Management repositories. 2023-08-26T13:14:46.783 INFO:teuthology.orchestra.run.smithi124.stdout:Unable to read consumer identity 2023-08-26T13:14:48.224 INFO:teuthology.orchestra.run.smithi124.stdout:Copr repo for python3-asyncssh owned by ceph 18 kB/s | 3.5 kB 00:00 2023-08-26T13:14:48.271 INFO:teuthology.orchestra.run.smithi031.stdout:Copr repo for python3-asyncssh owned by ceph 19 kB/s | 3.5 kB 00:00 2023-08-26T13:14:49.408 INFO:teuthology.orchestra.run.smithi124.stdout:Extra Packages for Enterprise Linux 14 MB/s | 16 MB 00:01 2023-08-26T13:14:49.413 INFO:teuthology.orchestra.run.smithi031.stdout:Extra Packages for Enterprise Linux 14 MB/s | 16 MB 00:01 2023-08-26T13:14:52.866 INFO:teuthology.orchestra.run.smithi124.stdout:lab-extras 507 kB/s | 24 kB 00:00 2023-08-26T13:14:53.072 INFO:teuthology.orchestra.run.smithi031.stdout:lab-extras 462 kB/s | 24 kB 00:00 2023-08-26T13:14:53.510 INFO:teuthology.orchestra.run.smithi124.stdout:Package kernel-4.18.0-372.9.1.el8.x86_64 is already installed. 2023-08-26T13:14:53.557 INFO:teuthology.orchestra.run.smithi124.stdout:Dependencies resolved. 2023-08-26T13:14:53.557 INFO:teuthology.orchestra.run.smithi124.stdout:Nothing to do. 2023-08-26T13:14:53.558 INFO:teuthology.orchestra.run.smithi124.stdout:Complete! 2023-08-26T13:14:53.662 DEBUG:teuthology.orchestra.run.smithi124:> echo no | sudo yum reinstall kernel || true 2023-08-26T13:14:53.824 INFO:teuthology.orchestra.run.smithi031.stdout:Package kernel-4.18.0-372.9.1.el8.x86_64 is already installed. 2023-08-26T13:14:53.874 INFO:teuthology.orchestra.run.smithi031.stdout:Dependencies resolved. 2023-08-26T13:14:53.876 INFO:teuthology.orchestra.run.smithi031.stdout:Nothing to do. 2023-08-26T13:14:53.876 INFO:teuthology.orchestra.run.smithi031.stdout:Complete! 2023-08-26T13:14:53.986 DEBUG:teuthology.orchestra.run.smithi031:> echo no | sudo yum reinstall kernel || true 2023-08-26T13:14:54.030 INFO:teuthology.orchestra.run.smithi124.stdout:Updating Subscription Management repositories. 2023-08-26T13:14:54.030 INFO:teuthology.orchestra.run.smithi124.stdout:Unable to read consumer identity 2023-08-26T13:14:54.296 INFO:teuthology.orchestra.run.smithi124.stdout:Last metadata expiration check: 0:00:02 ago on Sat 26 Aug 2023 01:14:52 PM UTC. 2023-08-26T13:14:54.334 INFO:teuthology.orchestra.run.smithi031.stdout:Updating Subscription Management repositories. 2023-08-26T13:14:54.334 INFO:teuthology.orchestra.run.smithi031.stdout:Unable to read consumer identity 2023-08-26T13:14:54.336 INFO:teuthology.orchestra.run.smithi124.stdout:Installed package kernel-4.18.0-372.9.1.el8.x86_64 (from anaconda) not available. 2023-08-26T13:14:54.383 INFO:teuthology.orchestra.run.smithi124.stderr:Error: No packages marked for reinstall. 2023-08-26T13:14:54.429 DEBUG:teuthology.orchestra.run.smithi124:> sudo yum reinstall -y kernel || true 2023-08-26T13:14:54.618 INFO:teuthology.orchestra.run.smithi031.stdout:Last metadata expiration check: 0:00:01 ago on Sat 26 Aug 2023 01:14:53 PM UTC. 2023-08-26T13:14:54.664 INFO:teuthology.orchestra.run.smithi031.stdout:Installed package kernel-4.18.0-372.9.1.el8.x86_64 (from anaconda) not available. 2023-08-26T13:14:54.694 INFO:teuthology.orchestra.run.smithi031.stderr:Error: No packages marked for reinstall. 2023-08-26T13:14:54.744 DEBUG:teuthology.orchestra.run.smithi031:> sudo yum reinstall -y kernel || true 2023-08-26T13:14:54.758 INFO:teuthology.orchestra.run.smithi124.stdout:Updating Subscription Management repositories. 2023-08-26T13:14:54.758 INFO:teuthology.orchestra.run.smithi124.stdout:Unable to read consumer identity 2023-08-26T13:14:55.016 INFO:teuthology.orchestra.run.smithi124.stdout:Last metadata expiration check: 0:00:03 ago on Sat 26 Aug 2023 01:14:52 PM UTC. 2023-08-26T13:14:55.057 INFO:teuthology.orchestra.run.smithi124.stdout:Installed package kernel-4.18.0-372.9.1.el8.x86_64 (from anaconda) not available. 2023-08-26T13:14:55.081 INFO:teuthology.orchestra.run.smithi124.stderr:Error: No packages marked for reinstall. 2023-08-26T13:14:55.094 INFO:teuthology.orchestra.run.smithi031.stdout:Updating Subscription Management repositories. 2023-08-26T13:14:55.094 INFO:teuthology.orchestra.run.smithi031.stdout:Unable to read consumer identity 2023-08-26T13:14:55.128 DEBUG:teuthology.orchestra.run.smithi124:> rpm -q kernel | sort -rV | head -n 1 2023-08-26T13:14:55.207 INFO:teuthology.orchestra.run.smithi124.stdout:kernel-4.18.0-372.9.1.el8.x86_64 2023-08-26T13:14:55.208 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 4.18.0-372.9.1.el8.x86_64 2023-08-26T13:14:55.208 INFO:teuthology.task.kernel:Newest distro kernel installed and running 2023-08-26T13:14:55.208 INFO:teuthology.task.kernel:Enabling kdb on cluster2.mon.a... 2023-08-26T13:14:55.209 DEBUG:teuthology.orchestra.run.smithi124:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2023-08-26T13:14:55.244 INFO:teuthology.orchestra.run.smithi124.stdout:ttyS1 2023-08-26T13:14:55.257 DEBUG:teuthology.parallel:result is None 2023-08-26T13:14:55.370 INFO:teuthology.orchestra.run.smithi031.stdout:Last metadata expiration check: 0:00:02 ago on Sat 26 Aug 2023 01:14:53 PM UTC. 2023-08-26T13:14:55.411 INFO:teuthology.orchestra.run.smithi031.stdout:Installed package kernel-4.18.0-372.9.1.el8.x86_64 (from anaconda) not available. 2023-08-26T13:14:55.440 INFO:teuthology.orchestra.run.smithi031.stderr:Error: No packages marked for reinstall. 2023-08-26T13:14:55.492 DEBUG:teuthology.orchestra.run.smithi031:> rpm -q kernel | sort -rV | head -n 1 2023-08-26T13:14:55.574 INFO:teuthology.orchestra.run.smithi031.stdout:kernel-4.18.0-372.9.1.el8.x86_64 2023-08-26T13:14:55.575 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 4.18.0-372.9.1.el8.x86_64 2023-08-26T13:14:55.575 INFO:teuthology.task.kernel:Newest distro kernel installed and running 2023-08-26T13:14:55.575 INFO:teuthology.task.kernel:Enabling kdb on cluster1.mon.a... 2023-08-26T13:14:55.576 DEBUG:teuthology.orchestra.run.smithi031:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2023-08-26T13:14:55.604 INFO:teuthology.orchestra.run.smithi031.stdout:ttyS1 2023-08-26T13:14:55.621 DEBUG:teuthology.parallel:result is None 2023-08-26T13:14:55.622 INFO:teuthology.run_tasks:Running task internal.base... 2023-08-26T13:14:55.635 INFO:teuthology.task.internal:Creating test directory... 2023-08-26T13:14:55.635 DEBUG:teuthology.orchestra.run.smithi031:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2023-08-26T13:14:55.664 DEBUG:teuthology.orchestra.run.smithi124:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2023-08-26T13:14:55.682 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2023-08-26T13:14:55.690 INFO:teuthology.run_tasks:Running task internal.archive... 2023-08-26T13:14:55.702 INFO:teuthology.task.internal:Creating archive directory... 2023-08-26T13:14:55.703 DEBUG:teuthology.orchestra.run.smithi031:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2023-08-26T13:14:55.721 DEBUG:teuthology.orchestra.run.smithi124:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2023-08-26T13:14:55.751 INFO:teuthology.run_tasks:Running task internal.coredump... 2023-08-26T13:14:55.760 INFO:teuthology.task.internal:Enabling coredump saving... 2023-08-26T13:14:55.761 DEBUG:teuthology.orchestra.run.smithi031:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2023-08-26T13:14:55.779 DEBUG:teuthology.orchestra.run.smithi124:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2023-08-26T13:14:55.807 INFO:teuthology.orchestra.run.smithi031.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-08-26T13:14:55.812 INFO:teuthology.orchestra.run.smithi124.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-08-26T13:14:55.818 INFO:teuthology.orchestra.run.smithi031.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-08-26T13:14:55.822 INFO:teuthology.orchestra.run.smithi124.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-08-26T13:14:55.824 INFO:teuthology.run_tasks:Running task internal.sudo... 2023-08-26T13:14:55.832 INFO:teuthology.task.internal:Configuring sudo... 2023-08-26T13:14:55.833 DEBUG:teuthology.orchestra.run.smithi031:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2023-08-26T13:14:55.862 DEBUG:teuthology.orchestra.run.smithi124:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2023-08-26T13:14:55.890 INFO:teuthology.run_tasks:Running task internal.syslog... 2023-08-26T13:14:55.900 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2023-08-26T13:14:55.901 DEBUG:teuthology.orchestra.run.smithi031:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2023-08-26T13:14:55.930 DEBUG:teuthology.orchestra.run.smithi124:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2023-08-26T13:14:55.946 DEBUG:teuthology.orchestra.run.smithi031:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2023-08-26T13:14:56.030 DEBUG:teuthology.orchestra.run.smithi031:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2023-08-26T13:14:56.079 DEBUG:teuthology.orchestra.run.smithi031:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2023-08-26T13:14:56.163 DEBUG:teuthology.orchestra.run.smithi031:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2023-08-26T13:14:56.204 DEBUG:teuthology.orchestra.run.smithi031:> set -ex 2023-08-26T13:14:56.205 DEBUG:teuthology.orchestra.run.smithi031:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2023-08-26T13:14:56.271 DEBUG:teuthology.orchestra.run.smithi124:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2023-08-26T13:14:56.315 DEBUG:teuthology.orchestra.run.smithi124:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2023-08-26T13:14:56.369 DEBUG:teuthology.orchestra.run.smithi124:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2023-08-26T13:14:56.453 DEBUG:teuthology.orchestra.run.smithi124:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2023-08-26T13:14:56.495 DEBUG:teuthology.orchestra.run.smithi124:> set -ex 2023-08-26T13:14:56.496 DEBUG:teuthology.orchestra.run.smithi124:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2023-08-26T13:14:56.563 DEBUG:teuthology.orchestra.run.smithi031:> sudo service rsyslog restart 2023-08-26T13:14:56.565 DEBUG:teuthology.orchestra.run.smithi124:> sudo service rsyslog restart 2023-08-26T13:14:56.617 INFO:teuthology.orchestra.run.smithi031.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2023-08-26T13:14:56.658 INFO:teuthology.orchestra.run.smithi124.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2023-08-26T13:14:57.009 INFO:teuthology.run_tasks:Running task internal.timer... 2023-08-26T13:14:57.016 INFO:teuthology.task.internal:Starting timer... 2023-08-26T13:14:57.017 INFO:teuthology.run_tasks:Running task pcp... 2023-08-26T13:14:57.032 INFO:teuthology.run_tasks:Running task selinux... 2023-08-26T13:14:57.045 DEBUG:teuthology.task:Applying overrides for task selinux: {'whitelist': ['scontext=system_u:system_r:logrotate_t:s0']} 2023-08-26T13:14:57.046 DEBUG:teuthology.orchestra.run.smithi031:> sudo service auditd rotate 2023-08-26T13:14:57.102 INFO:teuthology.orchestra.run.smithi031.stdout:Rotating logs: 2023-08-26T13:14:57.104 DEBUG:teuthology.orchestra.run.smithi124:> sudo service auditd rotate 2023-08-26T13:14:57.167 INFO:teuthology.orchestra.run.smithi124.stdout:Rotating logs: 2023-08-26T13:14:57.169 DEBUG:teuthology.task.selinux:Getting current SELinux state 2023-08-26T13:14:57.169 DEBUG:teuthology.orchestra.run.smithi031:> /usr/sbin/getenforce 2023-08-26T13:14:57.190 INFO:teuthology.orchestra.run.smithi031.stdout:Permissive 2023-08-26T13:14:57.191 DEBUG:teuthology.orchestra.run.smithi124:> /usr/sbin/getenforce 2023-08-26T13:14:57.217 INFO:teuthology.orchestra.run.smithi124.stdout:Permissive 2023-08-26T13:14:57.218 DEBUG:teuthology.task.selinux:Existing SELinux modes: {'ubuntu@smithi031.front.sepia.ceph.com': 'permissive', 'ubuntu@smithi124.front.sepia.ceph.com': 'permissive'} 2023-08-26T13:14:57.219 DEBUG:teuthology.orchestra.run.smithi031:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2023-08-26T13:14:57.260 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-08-26T13:14:57.261 DEBUG:teuthology.orchestra.run.smithi124:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2023-08-26T13:14:57.288 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-08-26T13:14:57.289 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2023-08-26T13:14:57.289 DEBUG:teuthology.orchestra.run.smithi031:> sudo /usr/sbin/setenforce permissive 2023-08-26T13:14:57.332 DEBUG:teuthology.orchestra.run.smithi124:> sudo /usr/sbin/setenforce permissive 2023-08-26T13:14:57.372 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2023-08-26T13:14:57.386 DEBUG:teuthology.repo_utils:Setting repo remote to https://git.ceph.com/ceph-cm-ansible.git 2023-08-26T13:14:57.392 INFO:teuthology.repo_utils:Fetching main from origin 2023-08-26T13:14:57.497 INFO:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2023-08-26T13:14:57.511 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2023-08-26T13:14:57.513 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi031.front.sepia.ceph.com,smithi124.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2023-08-26T13:25:24.141 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi031.front.sepia.ceph.com'), Remote(name='ubuntu@smithi124.front.sepia.ceph.com')] 2023-08-26T13:25:24.143 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi031.front.sepia.ceph.com' 2023-08-26T13:25:24.144 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi031.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-08-26T13:25:24.212 DEBUG:teuthology.orchestra.run.smithi031:> true 2023-08-26T13:25:24.281 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi031.front.sepia.ceph.com' 2023-08-26T13:25:24.281 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi124.front.sepia.ceph.com' 2023-08-26T13:25:24.282 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi124.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-08-26T13:25:24.351 DEBUG:teuthology.orchestra.run.smithi124:> true 2023-08-26T13:25:24.416 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi124.front.sepia.ceph.com' 2023-08-26T13:25:24.417 INFO:teuthology.run_tasks:Running task clock... 2023-08-26T13:25:24.427 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2023-08-26T13:25:24.428 INFO:teuthology.orchestra.run:Running command with timeout 360 2023-08-26T13:25:24.428 DEBUG:teuthology.orchestra.run.smithi031:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2023-08-26T13:25:24.431 INFO:teuthology.orchestra.run:Running command with timeout 360 2023-08-26T13:25:24.431 DEBUG:teuthology.orchestra.run.smithi124:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2023-08-26T13:25:24.462 INFO:teuthology.orchestra.run.smithi031.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2023-08-26T13:25:24.477 INFO:teuthology.orchestra.run.smithi031.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2023-08-26T13:25:24.489 INFO:teuthology.orchestra.run.smithi124.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2023-08-26T13:25:24.503 INFO:teuthology.orchestra.run.smithi124.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2023-08-26T13:25:24.544 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: ntpd: command not found 2023-08-26T13:25:24.556 INFO:teuthology.orchestra.run.smithi031.stdout:506 Cannot talk to daemon 2023-08-26T13:25:24.572 INFO:teuthology.orchestra.run.smithi031.stderr:Failed to start ntp.service: Unit ntp.service not found. 2023-08-26T13:25:24.573 INFO:teuthology.orchestra.run.smithi124.stderr:sudo: ntpd: command not found 2023-08-26T13:25:24.585 INFO:teuthology.orchestra.run.smithi124.stdout:506 Cannot talk to daemon 2023-08-26T13:25:24.586 INFO:teuthology.orchestra.run.smithi031.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2023-08-26T13:25:24.599 INFO:teuthology.orchestra.run.smithi124.stderr:Failed to start ntp.service: Unit ntp.service not found. 2023-08-26T13:25:24.613 INFO:teuthology.orchestra.run.smithi124.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2023-08-26T13:25:24.652 INFO:teuthology.orchestra.run.smithi031.stderr:bash: ntpq: command not found 2023-08-26T13:25:24.656 INFO:teuthology.orchestra.run.smithi031.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2023-08-26T13:25:24.656 INFO:teuthology.orchestra.run.smithi031.stdout:=============================================================================== 2023-08-26T13:25:24.656 INFO:teuthology.orchestra.run.smithi031.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-08-26T13:25:24.656 INFO:teuthology.orchestra.run.smithi031.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-08-26T13:25:24.657 INFO:teuthology.orchestra.run.smithi031.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-08-26T13:25:24.657 INFO:teuthology.orchestra.run.smithi031.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-08-26T13:25:24.683 INFO:teuthology.orchestra.run.smithi124.stderr:bash: ntpq: command not found 2023-08-26T13:25:24.687 INFO:teuthology.orchestra.run.smithi124.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2023-08-26T13:25:24.687 INFO:teuthology.orchestra.run.smithi124.stdout:=============================================================================== 2023-08-26T13:25:24.687 INFO:teuthology.orchestra.run.smithi124.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-08-26T13:25:24.687 INFO:teuthology.orchestra.run.smithi124.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-08-26T13:25:24.687 INFO:teuthology.orchestra.run.smithi124.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-08-26T13:25:24.688 INFO:teuthology.orchestra.run.smithi124.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-08-26T13:25:24.689 INFO:teuthology.run_tasks:Running task install... 2023-08-26T13:25:24.697 DEBUG:teuthology.task.install:project ceph 2023-08-26T13:25:24.698 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'flavor': 'default', 'sha1': 'e7eb9853fddaf661c68f207346131a7e00d8ed2f'}} 2023-08-26T13:25:24.698 DEBUG:teuthology.task.install:config {'extra_packages': ['rbd-mirror'], 'flavor': 'default', 'sha1': 'e7eb9853fddaf661c68f207346131a7e00d8ed2f'} 2023-08-26T13:25:24.698 INFO:teuthology.task.install:Using flavor: default 2023-08-26T13:25:24.704 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'ceph-volume', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-fuse', 'ceph-volume', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd']} 2023-08-26T13:25:24.704 INFO:teuthology.task.install:extra packages: ['rbd-mirror'] 2023-08-26T13:25:24.705 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': ['rbd-mirror'], 'extra_system_packages': [], 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': 'e7eb9853fddaf661c68f207346131a7e00d8ed2f', 'tag': None, 'wait_for_package': False} 2023-08-26T13:25:24.705 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&sha1=e7eb9853fddaf661c68f207346131a7e00d8ed2f 2023-08-26T13:25:24.707 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': ['rbd-mirror'], 'extra_system_packages': [], 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': 'e7eb9853fddaf661c68f207346131a7e00d8ed2f', 'tag': None, 'wait_for_package': False} 2023-08-26T13:25:24.708 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&sha1=e7eb9853fddaf661c68f207346131a7e00d8ed2f 2023-08-26T13:25:24.845 INFO:teuthology.task.install.rpm:Pulling from https://1.chacra.ceph.com/r/ceph/main/e7eb9853fddaf661c68f207346131a7e00d8ed2f/centos/8/flavors/default/ 2023-08-26T13:25:24.845 INFO:teuthology.task.install.rpm:Package version is 18.0.0-5706.ge7eb9853 2023-08-26T13:25:25.003 INFO:teuthology.task.install.rpm:Pulling from https://1.chacra.ceph.com/r/ceph/main/e7eb9853fddaf661c68f207346131a7e00d8ed2f/centos/8/flavors/default/ 2023-08-26T13:25:25.003 INFO:teuthology.task.install.rpm:Package version is 18.0.0-5706.ge7eb9853 2023-08-26T13:25:25.049 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://1.chacra.ceph.com/r/ceph/main/e7eb9853fddaf661c68f207346131a7e00d8ed2f/centos/8/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://1.chacra.ceph.com/r/ceph/main/e7eb9853fddaf661c68f207346131a7e00d8ed2f/centos/8/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://1.chacra.ceph.com/r/ceph/main/e7eb9853fddaf661c68f207346131a7e00d8ed2f/centos/8/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2023-08-26T13:25:25.049 DEBUG:teuthology.orchestra.run.smithi124:> set -ex 2023-08-26T13:25:25.049 DEBUG:teuthology.orchestra.run.smithi124:> sudo dd of=/etc/yum.repos.d/ceph.repo 2023-08-26T13:25:25.083 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd, rbd-mirror on remote rpm x86_64 2023-08-26T13:25:25.083 DEBUG:teuthology.orchestra.run.smithi124:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/e7eb9853fddaf661c68f207346131a7e00d8ed2f/;g' /etc/yum.repos.d/ceph.repo ; fi 2023-08-26T13:25:25.152 DEBUG:teuthology.orchestra.run.smithi124:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2023-08-26T13:25:25.174 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://1.chacra.ceph.com/r/ceph/main/e7eb9853fddaf661c68f207346131a7e00d8ed2f/centos/8/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://1.chacra.ceph.com/r/ceph/main/e7eb9853fddaf661c68f207346131a7e00d8ed2f/centos/8/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://1.chacra.ceph.com/r/ceph/main/e7eb9853fddaf661c68f207346131a7e00d8ed2f/centos/8/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2023-08-26T13:25:25.175 DEBUG:teuthology.orchestra.run.smithi031:> set -ex 2023-08-26T13:25:25.175 DEBUG:teuthology.orchestra.run.smithi031:> sudo dd of=/etc/yum.repos.d/ceph.repo 2023-08-26T13:25:25.205 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd, rbd-mirror on remote rpm x86_64 2023-08-26T13:25:25.205 DEBUG:teuthology.orchestra.run.smithi031:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/e7eb9853fddaf661c68f207346131a7e00d8ed2f/;g' /etc/yum.repos.d/ceph.repo ; fi 2023-08-26T13:25:25.229 DEBUG:teuthology.orchestra.run.smithi124:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2023-08-26T13:25:25.275 DEBUG:teuthology.orchestra.run.smithi031:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2023-08-26T13:25:25.295 INFO:teuthology.orchestra.run.smithi124.stdout:check_obsoletes = 1 2023-08-26T13:25:25.297 DEBUG:teuthology.orchestra.run.smithi124:> sudo yum clean all 2023-08-26T13:25:25.355 DEBUG:teuthology.orchestra.run.smithi031:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2023-08-26T13:25:25.422 INFO:teuthology.orchestra.run.smithi031.stdout:check_obsoletes = 1 2023-08-26T13:25:25.424 DEBUG:teuthology.orchestra.run.smithi031:> sudo yum clean all 2023-08-26T13:25:25.666 INFO:teuthology.orchestra.run.smithi124.stdout:Updating Subscription Management repositories. 2023-08-26T13:25:25.813 INFO:teuthology.orchestra.run.smithi031.stdout:Updating Subscription Management repositories. 2023-08-26T13:25:26.619 INFO:teuthology.orchestra.run.smithi124.stdout:44 files removed 2023-08-26T13:25:26.671 DEBUG:teuthology.orchestra.run.smithi124:> sudo yum -y install ceph-radosgw 2023-08-26T13:25:26.845 INFO:teuthology.orchestra.run.smithi031.stdout:44 files removed 2023-08-26T13:25:26.900 DEBUG:teuthology.orchestra.run.smithi031:> sudo yum -y install ceph-radosgw 2023-08-26T13:25:27.000 INFO:teuthology.orchestra.run.smithi124.stdout:Updating Subscription Management repositories. 2023-08-26T13:25:27.252 INFO:teuthology.orchestra.run.smithi031.stdout:Updating Subscription Management repositories. 2023-08-26T13:25:28.483 INFO:teuthology.orchestra.run.smithi124.stdout:ceph packages for x86_64 203 kB/s | 81 kB 00:00 2023-08-26T13:25:28.938 INFO:teuthology.orchestra.run.smithi031.stdout:ceph packages for x86_64 153 kB/s | 81 kB 00:00 2023-08-26T13:25:28.967 INFO:teuthology.orchestra.run.smithi124.stdout:ceph noarch packages 21 kB/s | 9.4 kB 00:00 2023-08-26T13:25:29.305 INFO:teuthology.orchestra.run.smithi031.stdout:ceph noarch packages 28 kB/s | 9.4 kB 00:00 2023-08-26T13:25:29.418 INFO:teuthology.orchestra.run.smithi124.stdout:ceph source packages 3.5 kB/s | 1.5 kB 00:00 2023-08-26T13:25:29.664 INFO:teuthology.orchestra.run.smithi031.stdout:ceph source packages 4.4 kB/s | 1.5 kB 00:00 2023-08-26T13:25:29.677 INFO:teuthology.orchestra.run.smithi124.stdout:Copr repo for python3-asyncssh owned by ceph 15 kB/s | 3.5 kB 00:00 2023-08-26T13:25:29.915 INFO:teuthology.orchestra.run.smithi031.stdout:Copr repo for python3-asyncssh owned by ceph 15 kB/s | 3.5 kB 00:00 2023-08-26T13:25:30.740 INFO:teuthology.orchestra.run.smithi031.stdout:Extra Packages for Enterprise Linux 20 MB/s | 16 MB 00:00 2023-08-26T13:25:31.621 INFO:teuthology.orchestra.run.smithi124.stdout:Extra Packages for Enterprise Linux 8.2 MB/s | 16 MB 00:01 2023-08-26T13:25:34.424 INFO:teuthology.orchestra.run.smithi031.stdout:lab-extras 432 kB/s | 24 kB 00:00 2023-08-26T13:25:35.078 INFO:teuthology.orchestra.run.smithi124.stdout:lab-extras 427 kB/s | 24 kB 00:00 2023-08-26T13:25:35.702 INFO:teuthology.orchestra.run.smithi031.stdout:Red Hat Enterprise Linux 8 for x86_64 - BaseOS 44 MB/s | 53 MB 00:01 2023-08-26T13:25:36.196 INFO:teuthology.orchestra.run.smithi124.stdout:Red Hat Enterprise Linux 8 for x86_64 - BaseOS 51 MB/s | 53 MB 00:01 2023-08-26T13:25:44.498 INFO:teuthology.orchestra.run.smithi124.stdout:Red Hat Enterprise Linux 8 for x86_64 - AppStre 45 MB/s | 47 MB 00:01 2023-08-26T13:25:44.694 INFO:teuthology.orchestra.run.smithi031.stdout:Red Hat Enterprise Linux 8 for x86_64 - AppStre 40 MB/s | 47 MB 00:01 2023-08-26T13:25:55.454 INFO:teuthology.orchestra.run.smithi124.stdout:Dependencies resolved. 2023-08-26T13:25:55.457 INFO:teuthology.orchestra.run.smithi124.stdout:==================================================================================================== 2023-08-26T13:25:55.457 INFO:teuthology.orchestra.run.smithi124.stdout: Package Arch Version Repository Size 2023-08-26T13:25:55.458 INFO:teuthology.orchestra.run.smithi124.stdout:==================================================================================================== 2023-08-26T13:25:55.458 INFO:teuthology.orchestra.run.smithi124.stdout:Installing: 2023-08-26T13:25:55.458 INFO:teuthology.orchestra.run.smithi124.stdout: ceph-radosgw x86_64 2:18.0.0-5706.ge7eb9853.el8 ceph 14 M 2023-08-26T13:25:55.458 INFO:teuthology.orchestra.run.smithi124.stdout:Upgrading: 2023-08-26T13:25:55.459 INFO:teuthology.orchestra.run.smithi124.stdout: librados2 x86_64 2:18.0.0-5706.ge7eb9853.el8 ceph 3.7 M 2023-08-26T13:25:55.459 INFO:teuthology.orchestra.run.smithi124.stdout: librbd1 x86_64 2:18.0.0-5706.ge7eb9853.el8 ceph 3.6 M 2023-08-26T13:25:55.459 INFO:teuthology.orchestra.run.smithi124.stdout:Installing dependencies: 2023-08-26T13:25:55.459 INFO:teuthology.orchestra.run.smithi124.stdout: ceph-base x86_64 2:18.0.0-5706.ge7eb9853.el8 ceph 5.4 M 2023-08-26T13:25:55.459 INFO:teuthology.orchestra.run.smithi124.stdout: ceph-common x86_64 2:18.0.0-5706.ge7eb9853.el8 ceph 22 M 2023-08-26T13:25:55.460 INFO:teuthology.orchestra.run.smithi124.stdout: ceph-selinux x86_64 2:18.0.0-5706.ge7eb9853.el8 ceph 24 k 2023-08-26T13:25:55.460 INFO:teuthology.orchestra.run.smithi124.stdout: gperftools-libs x86_64 1:2.7-9.el8 epel 306 k 2023-08-26T13:25:55.460 INFO:teuthology.orchestra.run.smithi124.stdout: libcephfs2 x86_64 2:18.0.0-5706.ge7eb9853.el8 ceph 784 k 2023-08-26T13:25:55.460 INFO:teuthology.orchestra.run.smithi124.stdout: liboath x86_64 2.6.2-3.el8 epel 59 k 2023-08-26T13:25:55.460 INFO:teuthology.orchestra.run.smithi124.stdout: librabbitmq x86_64 0.9.0-3.el8 rhel-8-for-x86_64-baseos-rpms 47 k 2023-08-26T13:25:55.461 INFO:teuthology.orchestra.run.smithi124.stdout: libradosstriper1 x86_64 2:18.0.0-5706.ge7eb9853.el8 ceph 472 k 2023-08-26T13:25:55.461 INFO:teuthology.orchestra.run.smithi124.stdout: librdkafka x86_64 0.11.4-3.el8 rhel-8-for-x86_64-appstream-rpms 354 k 2023-08-26T13:25:55.461 INFO:teuthology.orchestra.run.smithi124.stdout: librgw2 x86_64 2:18.0.0-5706.ge7eb9853.el8 ceph 6.8 M 2023-08-26T13:25:55.461 INFO:teuthology.orchestra.run.smithi124.stdout: libunwind x86_64 1.3.1-3.el8 epel 75 k 2023-08-26T13:25:55.461 INFO:teuthology.orchestra.run.smithi124.stdout: lttng-ust x86_64 2.8.1-11.el8 rhel-8-for-x86_64-appstream-rpms 259 k 2023-08-26T13:25:55.461 INFO:teuthology.orchestra.run.smithi124.stdout: python3-ceph-argparse x86_64 2:18.0.0-5706.ge7eb9853.el8 ceph 45 k 2023-08-26T13:25:55.462 INFO:teuthology.orchestra.run.smithi124.stdout: python3-ceph-common x86_64 2:18.0.0-5706.ge7eb9853.el8 ceph 125 k 2023-08-26T13:25:55.462 INFO:teuthology.orchestra.run.smithi124.stdout: python3-cephfs x86_64 2:18.0.0-5706.ge7eb9853.el8 ceph 197 k 2023-08-26T13:25:55.462 INFO:teuthology.orchestra.run.smithi124.stdout: python3-prettytable noarch 0.7.2-14.el8 rhel-8-for-x86_64-appstream-rpms 44 k 2023-08-26T13:25:55.462 INFO:teuthology.orchestra.run.smithi124.stdout: python3-rados x86_64 2:18.0.0-5706.ge7eb9853.el8 ceph 371 k 2023-08-26T13:25:55.462 INFO:teuthology.orchestra.run.smithi124.stdout: python3-rbd x86_64 2:18.0.0-5706.ge7eb9853.el8 ceph 365 k 2023-08-26T13:25:55.463 INFO:teuthology.orchestra.run.smithi124.stdout: python3-rgw x86_64 2:18.0.0-5706.ge7eb9853.el8 ceph 107 k 2023-08-26T13:25:55.463 INFO:teuthology.orchestra.run.smithi124.stdout: thrift x86_64 0.13.0-2.el8 epel 1.7 M 2023-08-26T13:25:55.463 INFO:teuthology.orchestra.run.smithi124.stdout: 2023-08-26T13:25:55.463 INFO:teuthology.orchestra.run.smithi124.stdout:Transaction Summary 2023-08-26T13:25:55.463 INFO:teuthology.orchestra.run.smithi124.stdout:==================================================================================================== 2023-08-26T13:25:55.463 INFO:teuthology.orchestra.run.smithi124.stdout:Install 21 Packages 2023-08-26T13:25:55.464 INFO:teuthology.orchestra.run.smithi124.stdout:Upgrade 2 Packages 2023-08-26T13:25:55.464 INFO:teuthology.orchestra.run.smithi124.stdout: 2023-08-26T13:25:55.464 INFO:teuthology.orchestra.run.smithi124.stdout:Total download size: 61 M 2023-08-26T13:25:55.465 INFO:teuthology.orchestra.run.smithi124.stdout:Downloading Packages: 2023-08-26T13:25:56.465 INFO:teuthology.orchestra.run.smithi124.stdout:(1/23): ceph-base-18.0.0-5706.ge7eb9853.el8.x86 8.7 MB/s | 5.4 MB 00:00 2023-08-26T13:25:56.515 INFO:teuthology.orchestra.run.smithi124.stdout:(2/23): ceph-selinux-18.0.0-5706.ge7eb9853.el8. 486 kB/s | 24 kB 00:00 2023-08-26T13:25:56.616 INFO:teuthology.orchestra.run.smithi124.stdout:(3/23): libcephfs2-18.0.0-5706.ge7eb9853.el8.x8 7.7 MB/s | 784 kB 00:00 2023-08-26T13:25:56.691 INFO:teuthology.orchestra.run.smithi124.stdout:(4/23): libradosstriper1-18.0.0-5706.ge7eb9853. 6.1 MB/s | 472 kB 00:00 2023-08-26T13:25:56.882 INFO:teuthology.orchestra.run.smithi031.stdout:Dependencies resolved. 2023-08-26T13:25:56.885 INFO:teuthology.orchestra.run.smithi031.stdout:==================================================================================================== 2023-08-26T13:25:56.885 INFO:teuthology.orchestra.run.smithi031.stdout: Package Arch Version Repository Size 2023-08-26T13:25:56.885 INFO:teuthology.orchestra.run.smithi031.stdout:==================================================================================================== 2023-08-26T13:25:56.885 INFO:teuthology.orchestra.run.smithi031.stdout:Installing: 2023-08-26T13:25:56.885 INFO:teuthology.orchestra.run.smithi031.stdout: ceph-radosgw x86_64 2:18.0.0-5706.ge7eb9853.el8 ceph 14 M 2023-08-26T13:25:56.886 INFO:teuthology.orchestra.run.smithi031.stdout:Upgrading: 2023-08-26T13:25:56.886 INFO:teuthology.orchestra.run.smithi031.stdout: librados2 x86_64 2:18.0.0-5706.ge7eb9853.el8 ceph 3.7 M 2023-08-26T13:25:56.886 INFO:teuthology.orchestra.run.smithi031.stdout: librbd1 x86_64 2:18.0.0-5706.ge7eb9853.el8 ceph 3.6 M 2023-08-26T13:25:56.886 INFO:teuthology.orchestra.run.smithi031.stdout:Installing dependencies: 2023-08-26T13:25:56.886 INFO:teuthology.orchestra.run.smithi031.stdout: ceph-base x86_64 2:18.0.0-5706.ge7eb9853.el8 ceph 5.4 M 2023-08-26T13:25:56.887 INFO:teuthology.orchestra.run.smithi031.stdout: ceph-common x86_64 2:18.0.0-5706.ge7eb9853.el8 ceph 22 M 2023-08-26T13:25:56.887 INFO:teuthology.orchestra.run.smithi031.stdout: ceph-selinux x86_64 2:18.0.0-5706.ge7eb9853.el8 ceph 24 k 2023-08-26T13:25:56.887 INFO:teuthology.orchestra.run.smithi031.stdout: gperftools-libs x86_64 1:2.7-9.el8 epel 306 k 2023-08-26T13:25:56.887 INFO:teuthology.orchestra.run.smithi031.stdout: libcephfs2 x86_64 2:18.0.0-5706.ge7eb9853.el8 ceph 784 k 2023-08-26T13:25:56.888 INFO:teuthology.orchestra.run.smithi031.stdout: liboath x86_64 2.6.2-3.el8 epel 59 k 2023-08-26T13:25:56.888 INFO:teuthology.orchestra.run.smithi031.stdout: librabbitmq x86_64 0.9.0-3.el8 rhel-8-for-x86_64-baseos-rpms 47 k 2023-08-26T13:25:56.888 INFO:teuthology.orchestra.run.smithi031.stdout: libradosstriper1 x86_64 2:18.0.0-5706.ge7eb9853.el8 ceph 472 k 2023-08-26T13:25:56.888 INFO:teuthology.orchestra.run.smithi031.stdout: librdkafka x86_64 0.11.4-3.el8 rhel-8-for-x86_64-appstream-rpms 354 k 2023-08-26T13:25:56.888 INFO:teuthology.orchestra.run.smithi031.stdout: librgw2 x86_64 2:18.0.0-5706.ge7eb9853.el8 ceph 6.8 M 2023-08-26T13:25:56.889 INFO:teuthology.orchestra.run.smithi031.stdout: libunwind x86_64 1.3.1-3.el8 epel 75 k 2023-08-26T13:25:56.889 INFO:teuthology.orchestra.run.smithi031.stdout: lttng-ust x86_64 2.8.1-11.el8 rhel-8-for-x86_64-appstream-rpms 259 k 2023-08-26T13:25:56.889 INFO:teuthology.orchestra.run.smithi031.stdout: python3-ceph-argparse x86_64 2:18.0.0-5706.ge7eb9853.el8 ceph 45 k 2023-08-26T13:25:56.889 INFO:teuthology.orchestra.run.smithi031.stdout: python3-ceph-common x86_64 2:18.0.0-5706.ge7eb9853.el8 ceph 125 k 2023-08-26T13:25:56.889 INFO:teuthology.orchestra.run.smithi031.stdout: python3-cephfs x86_64 2:18.0.0-5706.ge7eb9853.el8 ceph 197 k 2023-08-26T13:25:56.890 INFO:teuthology.orchestra.run.smithi031.stdout: python3-prettytable noarch 0.7.2-14.el8 rhel-8-for-x86_64-appstream-rpms 44 k 2023-08-26T13:25:56.890 INFO:teuthology.orchestra.run.smithi031.stdout: python3-rados x86_64 2:18.0.0-5706.ge7eb9853.el8 ceph 371 k 2023-08-26T13:25:56.890 INFO:teuthology.orchestra.run.smithi031.stdout: python3-rbd x86_64 2:18.0.0-5706.ge7eb9853.el8 ceph 365 k 2023-08-26T13:25:56.890 INFO:teuthology.orchestra.run.smithi031.stdout: python3-rgw x86_64 2:18.0.0-5706.ge7eb9853.el8 ceph 107 k 2023-08-26T13:25:56.890 INFO:teuthology.orchestra.run.smithi031.stdout: thrift x86_64 0.13.0-2.el8 epel 1.7 M 2023-08-26T13:25:56.891 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-08-26T13:25:56.891 INFO:teuthology.orchestra.run.smithi031.stdout:Transaction Summary 2023-08-26T13:25:56.891 INFO:teuthology.orchestra.run.smithi031.stdout:==================================================================================================== 2023-08-26T13:25:56.891 INFO:teuthology.orchestra.run.smithi031.stdout:Install 21 Packages 2023-08-26T13:25:56.891 INFO:teuthology.orchestra.run.smithi031.stdout:Upgrade 2 Packages 2023-08-26T13:25:56.892 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-08-26T13:25:56.892 INFO:teuthology.orchestra.run.smithi031.stdout:Total download size: 61 M 2023-08-26T13:25:56.892 INFO:teuthology.orchestra.run.smithi031.stdout:Downloading Packages: 2023-08-26T13:25:57.050 INFO:teuthology.orchestra.run.smithi124.stdout:(5/23): ceph-radosgw-18.0.0-5706.ge7eb9853.el8. 11 MB/s | 14 MB 00:01 2023-08-26T13:25:57.109 INFO:teuthology.orchestra.run.smithi124.stdout:(6/23): python3-ceph-argparse-18.0.0-5706.ge7eb 782 kB/s | 45 kB 00:00 2023-08-26T13:25:57.176 INFO:teuthology.orchestra.run.smithi124.stdout:(7/23): python3-ceph-common-18.0.0-5706.ge7eb98 1.8 MB/s | 125 kB 00:00 2023-08-26T13:25:57.284 INFO:teuthology.orchestra.run.smithi124.stdout:(8/23): librgw2-18.0.0-5706.ge7eb9853.el8.x86_6 11 MB/s | 6.8 MB 00:00 2023-08-26T13:25:57.309 INFO:teuthology.orchestra.run.smithi124.stdout:(9/23): python3-cephfs-18.0.0-5706.ge7eb9853.el 1.4 MB/s | 197 kB 00:00 2023-08-26T13:25:57.360 INFO:teuthology.orchestra.run.smithi124.stdout:(10/23): python3-rados-18.0.0-5706.ge7eb9853.el 4.8 MB/s | 371 kB 00:00 2023-08-26T13:25:57.385 INFO:teuthology.orchestra.run.smithi124.stdout:(11/23): python3-rbd-18.0.0-5706.ge7eb9853.el8. 4.7 MB/s | 365 kB 00:00 2023-08-26T13:25:57.402 INFO:teuthology.orchestra.run.smithi124.stdout:(12/23): python3-rgw-18.0.0-5706.ge7eb9853.el8. 2.5 MB/s | 107 kB 00:00 2023-08-26T13:25:57.794 INFO:teuthology.orchestra.run.smithi124.stdout:(13/23): liboath-2.6.2-3.el8.x86_64.rpm 150 kB/s | 59 kB 00:00 2023-08-26T13:25:58.036 INFO:teuthology.orchestra.run.smithi124.stdout:(14/23): gperftools-libs-2.7-9.el8.x86_64.rpm 470 kB/s | 306 kB 00:00 2023-08-26T13:25:58.123 INFO:teuthology.orchestra.run.smithi124.stdout:(15/23): libunwind-1.3.1-3.el8.x86_64.rpm 281 kB/s | 75 kB 00:00 2023-08-26T13:25:58.229 INFO:teuthology.orchestra.run.smithi124.stdout:(16/23): librabbitmq-0.9.0-3.el8.x86_64.rpm 281 kB/s | 47 kB 00:00 2023-08-26T13:25:58.388 INFO:teuthology.orchestra.run.smithi124.stdout:(17/23): lttng-ust-2.8.1-11.el8.x86_64.rpm 1.6 MB/s | 259 kB 00:00 2023-08-26T13:25:58.504 INFO:teuthology.orchestra.run.smithi124.stdout:(18/23): python3-prettytable-0.7.2-14.el8.noarc 377 kB/s | 44 kB 00:00 2023-08-26T13:25:58.663 INFO:teuthology.orchestra.run.smithi124.stdout:(19/23): librdkafka-0.11.4-3.el8.x86_64.rpm 2.2 MB/s | 354 kB 00:00 2023-08-26T13:25:58.860 INFO:teuthology.orchestra.run.smithi031.stdout:(1/23): ceph-base-18.0.0-5706.ge7eb9853.el8.x86 3.0 MB/s | 5.4 MB 00:01 2023-08-26T13:25:58.985 INFO:teuthology.orchestra.run.smithi031.stdout:(2/23): ceph-selinux-18.0.0-5706.ge7eb9853.el8. 195 kB/s | 24 kB 00:00 2023-08-26T13:25:59.064 INFO:teuthology.orchestra.run.smithi124.stdout:(20/23): librados2-18.0.0-5706.ge7eb9853.el8.x8 9.2 MB/s | 3.7 MB 00:00 2023-08-26T13:25:59.194 INFO:teuthology.orchestra.run.smithi031.stdout:(3/23): libcephfs2-18.0.0-5706.ge7eb9853.el8.x8 3.7 MB/s | 784 kB 00:00 2023-08-26T13:25:59.411 INFO:teuthology.orchestra.run.smithi031.stdout:(4/23): libradosstriper1-18.0.0-5706.ge7eb9853. 2.1 MB/s | 472 kB 00:00 2023-08-26T13:25:59.430 INFO:teuthology.orchestra.run.smithi124.stdout:(21/23): librbd1-18.0.0-5706.ge7eb9853.el8.x86_ 9.9 MB/s | 3.6 MB 00:00 2023-08-26T13:25:59.831 INFO:teuthology.orchestra.run.smithi124.stdout:(22/23): thrift-0.13.0-2.el8.x86_64.rpm 990 kB/s | 1.7 MB 00:01 2023-08-26T13:25:59.919 INFO:teuthology.orchestra.run.smithi031.stdout:(5/23): ceph-radosgw-18.0.0-5706.ge7eb9853.el8. 4.7 MB/s | 14 MB 00:02 2023-08-26T13:26:00.044 INFO:teuthology.orchestra.run.smithi031.stdout:(6/23): python3-ceph-argparse-18.0.0-5706.ge7eb 358 kB/s | 45 kB 00:00 2023-08-26T13:26:00.145 INFO:teuthology.orchestra.run.smithi031.stdout:(7/23): python3-ceph-common-18.0.0-5706.ge7eb98 1.2 MB/s | 125 kB 00:00 2023-08-26T13:26:00.253 INFO:teuthology.orchestra.run.smithi031.stdout:(8/23): python3-cephfs-18.0.0-5706.ge7eb9853.el 1.8 MB/s | 197 kB 00:00 2023-08-26T13:26:00.395 INFO:teuthology.orchestra.run.smithi031.stdout:(9/23): librgw2-18.0.0-5706.ge7eb9853.el8.x86_6 6.9 MB/s | 6.8 MB 00:00 2023-08-26T13:26:00.420 INFO:teuthology.orchestra.run.smithi031.stdout:(10/23): python3-rados-18.0.0-5706.ge7eb9853.el 2.2 MB/s | 371 kB 00:00 2023-08-26T13:26:00.487 INFO:teuthology.orchestra.run.smithi031.stdout:(11/23): python3-rbd-18.0.0-5706.ge7eb9853.el8. 3.9 MB/s | 365 kB 00:00 2023-08-26T13:26:00.512 INFO:teuthology.orchestra.run.smithi031.stdout:(12/23): python3-rgw-18.0.0-5706.ge7eb9853.el8. 1.1 MB/s | 107 kB 00:00 2023-08-26T13:26:00.821 INFO:teuthology.orchestra.run.smithi031.stdout:(13/23): ceph-common-18.0.0-5706.ge7eb9853.el8. 5.9 MB/s | 22 MB 00:03 2023-08-26T13:26:00.878 INFO:teuthology.orchestra.run.smithi031.stdout:(14/23): liboath-2.6.2-3.el8.x86_64.rpm 161 kB/s | 59 kB 00:00 2023-08-26T13:26:01.005 INFO:teuthology.orchestra.run.smithi031.stdout:(15/23): gperftools-libs-2.7-9.el8.x86_64.rpm 592 kB/s | 306 kB 00:00 2023-08-26T13:26:01.072 INFO:teuthology.orchestra.run.smithi031.stdout:(16/23): libunwind-1.3.1-3.el8.x86_64.rpm 299 kB/s | 75 kB 00:00 2023-08-26T13:26:01.172 INFO:teuthology.orchestra.run.smithi031.stdout:(17/23): librabbitmq-0.9.0-3.el8.x86_64.rpm 280 kB/s | 47 kB 00:00 2023-08-26T13:26:01.214 INFO:teuthology.orchestra.run.smithi031.stdout:(18/23): thrift-0.13.0-2.el8.x86_64.rpm 5.2 MB/s | 1.7 MB 00:00 2023-08-26T13:26:01.248 INFO:teuthology.orchestra.run.smithi031.stdout:(19/23): lttng-ust-2.8.1-11.el8.x86_64.rpm 1.4 MB/s | 259 kB 00:00 2023-08-26T13:26:01.373 INFO:teuthology.orchestra.run.smithi031.stdout:(20/23): python3-prettytable-0.7.2-14.el8.noarc 219 kB/s | 44 kB 00:00 2023-08-26T13:26:01.407 INFO:teuthology.orchestra.run.smithi031.stdout:(21/23): librdkafka-0.11.4-3.el8.x86_64.rpm 1.8 MB/s | 354 kB 00:00 2023-08-26T13:26:01.598 INFO:teuthology.orchestra.run.smithi124.stdout:(23/23): ceph-common-18.0.0-5706.ge7eb9853.el8. 3.9 MB/s | 22 MB 00:05 2023-08-26T13:26:01.600 INFO:teuthology.orchestra.run.smithi124.stdout:-------------------------------------------------------------------------------- 2023-08-26T13:26:01.600 INFO:teuthology.orchestra.run.smithi124.stdout:Total 9.9 MB/s | 61 MB 00:06 2023-08-26T13:26:01.647 INFO:teuthology.orchestra.run.smithi124.stdout:Running transaction check 2023-08-26T13:26:01.691 INFO:teuthology.orchestra.run.smithi124.stdout:Transaction check succeeded. 2023-08-26T13:26:01.692 INFO:teuthology.orchestra.run.smithi124.stdout:Running transaction test 2023-08-26T13:26:01.799 INFO:teuthology.orchestra.run.smithi031.stdout:(22/23): librados2-18.0.0-5706.ge7eb9853.el8.x8 6.8 MB/s | 3.7 MB 00:00 2023-08-26T13:26:01.857 INFO:teuthology.orchestra.run.smithi031.stdout:(23/23): librbd1-18.0.0-5706.ge7eb9853.el8.x86_ 7.5 MB/s | 3.6 MB 00:00 2023-08-26T13:26:01.859 INFO:teuthology.orchestra.run.smithi031.stdout:-------------------------------------------------------------------------------- 2023-08-26T13:26:01.860 INFO:teuthology.orchestra.run.smithi031.stdout:Total 12 MB/s | 61 MB 00:04 2023-08-26T13:26:01.909 INFO:teuthology.orchestra.run.smithi031.stdout:Running transaction check 2023-08-26T13:26:01.958 INFO:teuthology.orchestra.run.smithi031.stdout:Transaction check succeeded. 2023-08-26T13:26:01.958 INFO:teuthology.orchestra.run.smithi031.stdout:Running transaction test 2023-08-26T13:26:02.027 INFO:teuthology.orchestra.run.smithi124.stdout:Transaction test succeeded. 2023-08-26T13:26:02.031 INFO:teuthology.orchestra.run.smithi124.stdout:Running transaction 2023-08-26T13:26:02.298 INFO:teuthology.orchestra.run.smithi031.stdout:Transaction test succeeded. 2023-08-26T13:26:02.302 INFO:teuthology.orchestra.run.smithi031.stdout:Running transaction 2023-08-26T13:26:03.476 INFO:teuthology.orchestra.run.smithi124.stdout: Preparing : 1/1 2023-08-26T13:26:03.512 INFO:teuthology.orchestra.run.smithi031.stdout: Preparing : 1/1 2023-08-26T13:26:03.714 INFO:teuthology.orchestra.run.smithi124.stdout: Installing : thrift-0.13.0-2.el8.x86_64 1/25 2023-08-26T13:26:03.768 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : thrift-0.13.0-2.el8.x86_64 1/25 2023-08-26T13:26:03.898 INFO:teuthology.orchestra.run.smithi124.stdout: Installing : lttng-ust-2.8.1-11.el8.x86_64 2/25 2023-08-26T13:26:03.959 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : lttng-ust-2.8.1-11.el8.x86_64 2/25 2023-08-26T13:26:04.306 INFO:teuthology.orchestra.run.smithi124.stdout: Running scriptlet: lttng-ust-2.8.1-11.el8.x86_64 2/25 2023-08-26T13:26:04.427 INFO:teuthology.orchestra.run.smithi031.stdout: Running scriptlet: lttng-ust-2.8.1-11.el8.x86_64 2/25 2023-08-26T13:26:04.506 INFO:teuthology.orchestra.run.smithi124.stdout: Upgrading : librados2-2:18.0.0-5706.ge7eb9853.el8.x86_64 3/25 2023-08-26T13:26:04.607 INFO:teuthology.orchestra.run.smithi124.stdout: Running scriptlet: librados2-2:18.0.0-5706.ge7eb9853.el8.x86_64 3/25 2023-08-26T13:26:04.799 INFO:teuthology.orchestra.run.smithi124.stdout: Installing : python3-rados-2:18.0.0-5706.ge7eb9853.el8.x86_64 4/25 2023-08-26T13:26:04.818 INFO:teuthology.orchestra.run.smithi031.stdout: Upgrading : librados2-2:18.0.0-5706.ge7eb9853.el8.x86_64 3/25 2023-08-26T13:26:04.919 INFO:teuthology.orchestra.run.smithi031.stdout: Running scriptlet: librados2-2:18.0.0-5706.ge7eb9853.el8.x86_64 3/25 2023-08-26T13:26:05.067 INFO:teuthology.orchestra.run.smithi124.stdout: Installing : liboath-2.6.2-3.el8.x86_64 5/25 2023-08-26T13:26:05.116 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : python3-rados-2:18.0.0-5706.ge7eb9853.el8.x86_64 4/25 2023-08-26T13:26:05.265 INFO:teuthology.orchestra.run.smithi124.stdout: Installing : libcephfs2-2:18.0.0-5706.ge7eb9853.el8.x86_64 6/25 2023-08-26T13:26:05.382 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : liboath-2.6.2-3.el8.x86_64 5/25 2023-08-26T13:26:05.584 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : libcephfs2-2:18.0.0-5706.ge7eb9853.el8.x86_64 6/25 2023-08-26T13:26:05.643 INFO:teuthology.orchestra.run.smithi124.stdout: Running scriptlet: libcephfs2-2:18.0.0-5706.ge7eb9853.el8.x86_64 6/25 2023-08-26T13:26:05.815 INFO:teuthology.orchestra.run.smithi124.stdout: Upgrading : librbd1-2:18.0.0-5706.ge7eb9853.el8.x86_64 7/25 2023-08-26T13:26:05.907 INFO:teuthology.orchestra.run.smithi124.stdout: Running scriptlet: librbd1-2:18.0.0-5706.ge7eb9853.el8.x86_64 7/25 2023-08-26T13:26:05.995 INFO:teuthology.orchestra.run.smithi031.stdout: Running scriptlet: libcephfs2-2:18.0.0-5706.ge7eb9853.el8.x86_64 6/25 2023-08-26T13:26:06.090 INFO:teuthology.orchestra.run.smithi124.stdout: Installing : librdkafka-0.11.4-3.el8.x86_64 8/25 2023-08-26T13:26:06.157 INFO:teuthology.orchestra.run.smithi124.stdout: Running scriptlet: librdkafka-0.11.4-3.el8.x86_64 8/25 2023-08-26T13:26:06.167 INFO:teuthology.orchestra.run.smithi031.stdout: Upgrading : librbd1-2:18.0.0-5706.ge7eb9853.el8.x86_64 7/25 2023-08-26T13:26:06.251 INFO:teuthology.orchestra.run.smithi031.stdout: Running scriptlet: librbd1-2:18.0.0-5706.ge7eb9853.el8.x86_64 7/25 2023-08-26T13:26:06.321 INFO:teuthology.orchestra.run.smithi124.stdout: Installing : librabbitmq-0.9.0-3.el8.x86_64 9/25 2023-08-26T13:26:06.426 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : librdkafka-0.11.4-3.el8.x86_64 8/25 2023-08-26T13:26:06.476 INFO:teuthology.orchestra.run.smithi124.stdout: Installing : python3-ceph-argparse-2:18.0.0-5706.ge7eb9853.el8. 10/25 2023-08-26T13:26:06.489 INFO:teuthology.orchestra.run.smithi031.stdout: Running scriptlet: librdkafka-0.11.4-3.el8.x86_64 8/25 2023-08-26T13:26:06.641 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : librabbitmq-0.9.0-3.el8.x86_64 9/25 2023-08-26T13:26:06.717 INFO:teuthology.orchestra.run.smithi124.stdout: Installing : python3-cephfs-2:18.0.0-5706.ge7eb9853.el8.x86_64 11/25 2023-08-26T13:26:06.813 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : python3-ceph-argparse-2:18.0.0-5706.ge7eb9853.el8. 10/25 2023-08-26T13:26:06.936 INFO:teuthology.orchestra.run.smithi124.stdout: Installing : python3-rbd-2:18.0.0-5706.ge7eb9853.el8.x86_64 12/25 2023-08-26T13:26:07.022 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : python3-cephfs-2:18.0.0-5706.ge7eb9853.el8.x86_64 11/25 2023-08-26T13:26:07.089 INFO:teuthology.orchestra.run.smithi124.stdout: Installing : libradosstriper1-2:18.0.0-5706.ge7eb9853.el8.x86_6 13/25 2023-08-26T13:26:07.189 INFO:teuthology.orchestra.run.smithi124.stdout: Running scriptlet: libradosstriper1-2:18.0.0-5706.ge7eb9853.el8.x86_6 13/25 2023-08-26T13:26:07.229 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : python3-rbd-2:18.0.0-5706.ge7eb9853.el8.x86_64 12/25 2023-08-26T13:26:07.349 INFO:teuthology.orchestra.run.smithi124.stdout: Installing : python3-prettytable-0.7.2-14.el8.noarch 14/25 2023-08-26T13:26:07.384 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : libradosstriper1-2:18.0.0-5706.ge7eb9853.el8.x86_6 13/25 2023-08-26T13:26:07.454 INFO:teuthology.orchestra.run.smithi031.stdout: Running scriptlet: libradosstriper1-2:18.0.0-5706.ge7eb9853.el8.x86_6 13/25 2023-08-26T13:26:07.547 INFO:teuthology.orchestra.run.smithi124.stdout: Installing : libunwind-1.3.1-3.el8.x86_64 15/25 2023-08-26T13:26:07.586 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : python3-prettytable-0.7.2-14.el8.noarch 14/25 2023-08-26T13:26:07.803 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : libunwind-1.3.1-3.el8.x86_64 15/25 2023-08-26T13:26:08.359 INFO:teuthology.orchestra.run.smithi124.stdout: Installing : gperftools-libs-1:2.7-9.el8.x86_64 16/25 2023-08-26T13:26:08.540 INFO:teuthology.orchestra.run.smithi124.stdout: Installing : librgw2-2:18.0.0-5706.ge7eb9853.el8.x86_64 17/25 2023-08-26T13:26:08.613 INFO:teuthology.orchestra.run.smithi124.stdout: Running scriptlet: librgw2-2:18.0.0-5706.ge7eb9853.el8.x86_64 17/25 2023-08-26T13:26:08.635 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : gperftools-libs-1:2.7-9.el8.x86_64 16/25 2023-08-26T13:26:08.825 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : librgw2-2:18.0.0-5706.ge7eb9853.el8.x86_64 17/25 2023-08-26T13:26:08.848 INFO:teuthology.orchestra.run.smithi124.stdout: Installing : python3-rgw-2:18.0.0-5706.ge7eb9853.el8.x86_64 18/25 2023-08-26T13:26:08.886 INFO:teuthology.orchestra.run.smithi031.stdout: Running scriptlet: librgw2-2:18.0.0-5706.ge7eb9853.el8.x86_64 17/25 2023-08-26T13:26:09.048 INFO:teuthology.orchestra.run.smithi124.stdout: Installing : python3-ceph-common-2:18.0.0-5706.ge7eb9853.el8.x8 19/25 2023-08-26T13:26:09.164 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : python3-rgw-2:18.0.0-5706.ge7eb9853.el8.x86_64 18/25 2023-08-26T13:26:09.342 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : python3-ceph-common-2:18.0.0-5706.ge7eb9853.el8.x8 19/25 2023-08-26T13:26:12.703 INFO:teuthology.orchestra.run.smithi124.stdout: Running scriptlet: ceph-common-2:18.0.0-5706.ge7eb9853.el8.x86_64 20/25 2023-08-26T13:26:13.214 INFO:teuthology.orchestra.run.smithi124.stdout: Installing : ceph-common-2:18.0.0-5706.ge7eb9853.el8.x86_64 20/25 2023-08-26T13:26:13.250 INFO:teuthology.orchestra.run.smithi031.stdout: Running scriptlet: ceph-common-2:18.0.0-5706.ge7eb9853.el8.x86_64 20/25 2023-08-26T13:26:13.516 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : ceph-common-2:18.0.0-5706.ge7eb9853.el8.x86_64 20/25 2023-08-26T13:26:13.806 INFO:teuthology.orchestra.run.smithi124.stdout: Running scriptlet: ceph-common-2:18.0.0-5706.ge7eb9853.el8.x86_64 20/25 2023-08-26T13:26:14.112 INFO:teuthology.orchestra.run.smithi124.stdout: Installing : ceph-base-2:18.0.0-5706.ge7eb9853.el8.x86_64 21/25 2023-08-26T13:26:14.164 INFO:teuthology.orchestra.run.smithi031.stdout: Running scriptlet: ceph-common-2:18.0.0-5706.ge7eb9853.el8.x86_64 20/25 2023-08-26T13:26:14.176 INFO:teuthology.orchestra.run.smithi124.stdout: Running scriptlet: ceph-base-2:18.0.0-5706.ge7eb9853.el8.x86_64 21/25 2023-08-26T13:26:14.347 INFO:teuthology.orchestra.run.smithi124.stdout: Installing : ceph-selinux-2:18.0.0-5706.ge7eb9853.el8.x86_64 22/25 2023-08-26T13:26:14.457 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : ceph-base-2:18.0.0-5706.ge7eb9853.el8.x86_64 21/25 2023-08-26T13:26:14.546 INFO:teuthology.orchestra.run.smithi031.stdout: Running scriptlet: ceph-base-2:18.0.0-5706.ge7eb9853.el8.x86_64 21/25 2023-08-26T13:26:14.725 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : ceph-selinux-2:18.0.0-5706.ge7eb9853.el8.x86_64 22/25 2023-08-26T13:26:33.024 INFO:teuthology.orchestra.run.smithi124.stdout: Running scriptlet: ceph-selinux-2:18.0.0-5706.ge7eb9853.el8.x86_64 22/25 2023-08-26T13:26:33.024 INFO:teuthology.orchestra.run.smithi124.stdout:skipping the directory /sys 2023-08-26T13:26:33.025 INFO:teuthology.orchestra.run.smithi124.stdout:skipping the directory /proc 2023-08-26T13:26:33.025 INFO:teuthology.orchestra.run.smithi124.stdout:skipping the directory /dev 2023-08-26T13:26:33.025 INFO:teuthology.orchestra.run.smithi124.stdout:skipping the directory /run 2023-08-26T13:26:33.025 INFO:teuthology.orchestra.run.smithi124.stdout:skipping the directory /mnt 2023-08-26T13:26:33.025 INFO:teuthology.orchestra.run.smithi124.stdout:skipping the directory /var/tmp 2023-08-26T13:26:33.026 INFO:teuthology.orchestra.run.smithi124.stdout:skipping the directory /home 2023-08-26T13:26:33.026 INFO:teuthology.orchestra.run.smithi124.stdout:skipping the directory /tmp 2023-08-26T13:26:33.026 INFO:teuthology.orchestra.run.smithi124.stdout:skipping the directory /dev 2023-08-26T13:26:33.026 INFO:teuthology.orchestra.run.smithi124.stdout: 2023-08-26T13:26:34.069 INFO:teuthology.orchestra.run.smithi031.stdout: Running scriptlet: ceph-selinux-2:18.0.0-5706.ge7eb9853.el8.x86_64 22/25 2023-08-26T13:26:34.070 INFO:teuthology.orchestra.run.smithi031.stdout:skipping the directory /sys 2023-08-26T13:26:34.070 INFO:teuthology.orchestra.run.smithi031.stdout:skipping the directory /proc 2023-08-26T13:26:34.070 INFO:teuthology.orchestra.run.smithi031.stdout:skipping the directory /dev 2023-08-26T13:26:34.070 INFO:teuthology.orchestra.run.smithi031.stdout:skipping the directory /run 2023-08-26T13:26:34.070 INFO:teuthology.orchestra.run.smithi031.stdout:skipping the directory /mnt 2023-08-26T13:26:34.071 INFO:teuthology.orchestra.run.smithi031.stdout:skipping the directory /var/tmp 2023-08-26T13:26:34.071 INFO:teuthology.orchestra.run.smithi031.stdout:skipping the directory /home 2023-08-26T13:26:34.071 INFO:teuthology.orchestra.run.smithi031.stdout:skipping the directory /tmp 2023-08-26T13:26:34.071 INFO:teuthology.orchestra.run.smithi031.stdout:skipping the directory /dev 2023-08-26T13:26:34.071 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-08-26T13:26:34.426 INFO:teuthology.orchestra.run.smithi124.stdout: Installing : ceph-radosgw-2:18.0.0-5706.ge7eb9853.el8.x86_64 23/25 2023-08-26T13:26:34.447 INFO:teuthology.orchestra.run.smithi124.stdout: Running scriptlet: ceph-radosgw-2:18.0.0-5706.ge7eb9853.el8.x86_64 23/25 2023-08-26T13:26:34.447 INFO:teuthology.orchestra.run.smithi124.stdout: Cleanup : librbd1-1:12.2.7-9.el8.x86_64 24/25 2023-08-26T13:26:34.688 INFO:teuthology.orchestra.run.smithi124.stdout: Running scriptlet: librbd1-1:12.2.7-9.el8.x86_64 24/25 2023-08-26T13:26:34.688 INFO:teuthology.orchestra.run.smithi124.stdout: Cleanup : librados2-1:12.2.7-9.el8.x86_64 25/25 2023-08-26T13:26:35.302 INFO:teuthology.orchestra.run.smithi124.stdout: Running scriptlet: librados2-1:12.2.7-9.el8.x86_64 25/25 2023-08-26T13:26:35.302 INFO:teuthology.orchestra.run.smithi124.stdout: Verifying : ceph-base-2:18.0.0-5706.ge7eb9853.el8.x86_64 1/25 2023-08-26T13:26:35.303 INFO:teuthology.orchestra.run.smithi124.stdout: Verifying : ceph-common-2:18.0.0-5706.ge7eb9853.el8.x86_64 2/25 2023-08-26T13:26:35.303 INFO:teuthology.orchestra.run.smithi124.stdout: Verifying : ceph-radosgw-2:18.0.0-5706.ge7eb9853.el8.x86_64 3/25 2023-08-26T13:26:35.303 INFO:teuthology.orchestra.run.smithi124.stdout: Verifying : ceph-selinux-2:18.0.0-5706.ge7eb9853.el8.x86_64 4/25 2023-08-26T13:26:35.303 INFO:teuthology.orchestra.run.smithi124.stdout: Verifying : libcephfs2-2:18.0.0-5706.ge7eb9853.el8.x86_64 5/25 2023-08-26T13:26:35.304 INFO:teuthology.orchestra.run.smithi124.stdout: Verifying : libradosstriper1-2:18.0.0-5706.ge7eb9853.el8.x86_6 6/25 2023-08-26T13:26:35.304 INFO:teuthology.orchestra.run.smithi124.stdout: Verifying : librgw2-2:18.0.0-5706.ge7eb9853.el8.x86_64 7/25 2023-08-26T13:26:35.304 INFO:teuthology.orchestra.run.smithi124.stdout: Verifying : python3-ceph-argparse-2:18.0.0-5706.ge7eb9853.el8. 8/25 2023-08-26T13:26:35.304 INFO:teuthology.orchestra.run.smithi124.stdout: Verifying : python3-ceph-common-2:18.0.0-5706.ge7eb9853.el8.x8 9/25 2023-08-26T13:26:35.304 INFO:teuthology.orchestra.run.smithi124.stdout: Verifying : python3-cephfs-2:18.0.0-5706.ge7eb9853.el8.x86_64 10/25 2023-08-26T13:26:35.305 INFO:teuthology.orchestra.run.smithi124.stdout: Verifying : python3-rados-2:18.0.0-5706.ge7eb9853.el8.x86_64 11/25 2023-08-26T13:26:35.305 INFO:teuthology.orchestra.run.smithi124.stdout: Verifying : python3-rbd-2:18.0.0-5706.ge7eb9853.el8.x86_64 12/25 2023-08-26T13:26:35.305 INFO:teuthology.orchestra.run.smithi124.stdout: Verifying : python3-rgw-2:18.0.0-5706.ge7eb9853.el8.x86_64 13/25 2023-08-26T13:26:35.305 INFO:teuthology.orchestra.run.smithi124.stdout: Verifying : gperftools-libs-1:2.7-9.el8.x86_64 14/25 2023-08-26T13:26:35.305 INFO:teuthology.orchestra.run.smithi124.stdout: Verifying : liboath-2.6.2-3.el8.x86_64 15/25 2023-08-26T13:26:35.306 INFO:teuthology.orchestra.run.smithi124.stdout: Verifying : libunwind-1.3.1-3.el8.x86_64 16/25 2023-08-26T13:26:35.306 INFO:teuthology.orchestra.run.smithi124.stdout: Verifying : thrift-0.13.0-2.el8.x86_64 17/25 2023-08-26T13:26:35.306 INFO:teuthology.orchestra.run.smithi124.stdout: Verifying : librabbitmq-0.9.0-3.el8.x86_64 18/25 2023-08-26T13:26:35.306 INFO:teuthology.orchestra.run.smithi124.stdout: Verifying : lttng-ust-2.8.1-11.el8.x86_64 19/25 2023-08-26T13:26:35.306 INFO:teuthology.orchestra.run.smithi124.stdout: Verifying : python3-prettytable-0.7.2-14.el8.noarch 20/25 2023-08-26T13:26:35.306 INFO:teuthology.orchestra.run.smithi124.stdout: Verifying : librdkafka-0.11.4-3.el8.x86_64 21/25 2023-08-26T13:26:35.307 INFO:teuthology.orchestra.run.smithi124.stdout: Verifying : librados2-2:18.0.0-5706.ge7eb9853.el8.x86_64 22/25 2023-08-26T13:26:35.307 INFO:teuthology.orchestra.run.smithi124.stdout: Verifying : librados2-1:12.2.7-9.el8.x86_64 23/25 2023-08-26T13:26:35.307 INFO:teuthology.orchestra.run.smithi124.stdout: Verifying : librbd1-2:18.0.0-5706.ge7eb9853.el8.x86_64 24/25 2023-08-26T13:26:35.554 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : ceph-radosgw-2:18.0.0-5706.ge7eb9853.el8.x86_64 23/25 2023-08-26T13:26:35.576 INFO:teuthology.orchestra.run.smithi031.stdout: Running scriptlet: ceph-radosgw-2:18.0.0-5706.ge7eb9853.el8.x86_64 23/25 2023-08-26T13:26:35.576 INFO:teuthology.orchestra.run.smithi031.stdout: Cleanup : librbd1-1:12.2.7-9.el8.x86_64 24/25 2023-08-26T13:26:35.765 INFO:teuthology.orchestra.run.smithi031.stdout: Running scriptlet: librbd1-1:12.2.7-9.el8.x86_64 24/25 2023-08-26T13:26:35.765 INFO:teuthology.orchestra.run.smithi031.stdout: Cleanup : librados2-1:12.2.7-9.el8.x86_64 25/25 2023-08-26T13:26:36.408 INFO:teuthology.orchestra.run.smithi031.stdout: Running scriptlet: librados2-1:12.2.7-9.el8.x86_64 25/25 2023-08-26T13:26:36.408 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : ceph-base-2:18.0.0-5706.ge7eb9853.el8.x86_64 1/25 2023-08-26T13:26:36.409 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : ceph-common-2:18.0.0-5706.ge7eb9853.el8.x86_64 2/25 2023-08-26T13:26:36.409 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : ceph-radosgw-2:18.0.0-5706.ge7eb9853.el8.x86_64 3/25 2023-08-26T13:26:36.409 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : ceph-selinux-2:18.0.0-5706.ge7eb9853.el8.x86_64 4/25 2023-08-26T13:26:36.409 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : libcephfs2-2:18.0.0-5706.ge7eb9853.el8.x86_64 5/25 2023-08-26T13:26:36.409 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : libradosstriper1-2:18.0.0-5706.ge7eb9853.el8.x86_6 6/25 2023-08-26T13:26:36.410 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : librgw2-2:18.0.0-5706.ge7eb9853.el8.x86_64 7/25 2023-08-26T13:26:36.410 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : python3-ceph-argparse-2:18.0.0-5706.ge7eb9853.el8. 8/25 2023-08-26T13:26:36.410 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : python3-ceph-common-2:18.0.0-5706.ge7eb9853.el8.x8 9/25 2023-08-26T13:26:36.410 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : python3-cephfs-2:18.0.0-5706.ge7eb9853.el8.x86_64 10/25 2023-08-26T13:26:36.410 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : python3-rados-2:18.0.0-5706.ge7eb9853.el8.x86_64 11/25 2023-08-26T13:26:36.411 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : python3-rbd-2:18.0.0-5706.ge7eb9853.el8.x86_64 12/25 2023-08-26T13:26:36.411 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : python3-rgw-2:18.0.0-5706.ge7eb9853.el8.x86_64 13/25 2023-08-26T13:26:36.411 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : gperftools-libs-1:2.7-9.el8.x86_64 14/25 2023-08-26T13:26:36.411 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : liboath-2.6.2-3.el8.x86_64 15/25 2023-08-26T13:26:36.411 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : libunwind-1.3.1-3.el8.x86_64 16/25 2023-08-26T13:26:36.412 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : thrift-0.13.0-2.el8.x86_64 17/25 2023-08-26T13:26:36.412 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : librabbitmq-0.9.0-3.el8.x86_64 18/25 2023-08-26T13:26:36.412 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : lttng-ust-2.8.1-11.el8.x86_64 19/25 2023-08-26T13:26:36.412 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : python3-prettytable-0.7.2-14.el8.noarch 20/25 2023-08-26T13:26:36.412 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : librdkafka-0.11.4-3.el8.x86_64 21/25 2023-08-26T13:26:36.413 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : librados2-2:18.0.0-5706.ge7eb9853.el8.x86_64 22/25 2023-08-26T13:26:36.413 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : librados2-1:12.2.7-9.el8.x86_64 23/25 2023-08-26T13:26:36.413 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : librbd1-2:18.0.0-5706.ge7eb9853.el8.x86_64 24/25 2023-08-26T13:26:36.694 INFO:teuthology.orchestra.run.smithi124.stdout: Verifying : librbd1-1:12.2.7-9.el8.x86_64 25/25 2023-08-26T13:26:36.695 INFO:teuthology.orchestra.run.smithi124.stdout:Last metadata expiration check: 0:00:53 ago on Sat 26 Aug 2023 01:25:43 PM UTC. 2023-08-26T13:26:37.886 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : librbd1-1:12.2.7-9.el8.x86_64 25/25 2023-08-26T13:26:37.887 INFO:teuthology.orchestra.run.smithi031.stdout:Last metadata expiration check: 0:00:54 ago on Sat 26 Aug 2023 01:25:43 PM UTC. 2023-08-26T13:26:45.159 INFO:teuthology.orchestra.run.smithi124.stdout:Installed products updated. 2023-08-26T13:26:45.354 INFO:teuthology.orchestra.run.smithi124.stdout: 2023-08-26T13:26:45.354 INFO:teuthology.orchestra.run.smithi124.stdout:Upgraded: 2023-08-26T13:26:45.355 INFO:teuthology.orchestra.run.smithi124.stdout: librados2-2:18.0.0-5706.ge7eb9853.el8.x86_64 2023-08-26T13:26:45.355 INFO:teuthology.orchestra.run.smithi124.stdout: librbd1-2:18.0.0-5706.ge7eb9853.el8.x86_64 2023-08-26T13:26:45.355 INFO:teuthology.orchestra.run.smithi124.stdout:Installed: 2023-08-26T13:26:45.355 INFO:teuthology.orchestra.run.smithi124.stdout: ceph-base-2:18.0.0-5706.ge7eb9853.el8.x86_64 2023-08-26T13:26:45.355 INFO:teuthology.orchestra.run.smithi124.stdout: ceph-common-2:18.0.0-5706.ge7eb9853.el8.x86_64 2023-08-26T13:26:45.356 INFO:teuthology.orchestra.run.smithi124.stdout: ceph-radosgw-2:18.0.0-5706.ge7eb9853.el8.x86_64 2023-08-26T13:26:45.356 INFO:teuthology.orchestra.run.smithi124.stdout: ceph-selinux-2:18.0.0-5706.ge7eb9853.el8.x86_64 2023-08-26T13:26:45.356 INFO:teuthology.orchestra.run.smithi124.stdout: gperftools-libs-1:2.7-9.el8.x86_64 2023-08-26T13:26:45.356 INFO:teuthology.orchestra.run.smithi124.stdout: libcephfs2-2:18.0.0-5706.ge7eb9853.el8.x86_64 2023-08-26T13:26:45.356 INFO:teuthology.orchestra.run.smithi124.stdout: liboath-2.6.2-3.el8.x86_64 2023-08-26T13:26:45.357 INFO:teuthology.orchestra.run.smithi124.stdout: librabbitmq-0.9.0-3.el8.x86_64 2023-08-26T13:26:45.357 INFO:teuthology.orchestra.run.smithi124.stdout: libradosstriper1-2:18.0.0-5706.ge7eb9853.el8.x86_64 2023-08-26T13:26:45.357 INFO:teuthology.orchestra.run.smithi124.stdout: librdkafka-0.11.4-3.el8.x86_64 2023-08-26T13:26:45.357 INFO:teuthology.orchestra.run.smithi124.stdout: librgw2-2:18.0.0-5706.ge7eb9853.el8.x86_64 2023-08-26T13:26:45.357 INFO:teuthology.orchestra.run.smithi124.stdout: libunwind-1.3.1-3.el8.x86_64 2023-08-26T13:26:45.358 INFO:teuthology.orchestra.run.smithi124.stdout: lttng-ust-2.8.1-11.el8.x86_64 2023-08-26T13:26:45.358 INFO:teuthology.orchestra.run.smithi124.stdout: python3-ceph-argparse-2:18.0.0-5706.ge7eb9853.el8.x86_64 2023-08-26T13:26:45.358 INFO:teuthology.orchestra.run.smithi124.stdout: python3-ceph-common-2:18.0.0-5706.ge7eb9853.el8.x86_64 2023-08-26T13:26:45.358 INFO:teuthology.orchestra.run.smithi124.stdout: python3-cephfs-2:18.0.0-5706.ge7eb9853.el8.x86_64 2023-08-26T13:26:45.358 INFO:teuthology.orchestra.run.smithi124.stdout: python3-prettytable-0.7.2-14.el8.noarch 2023-08-26T13:26:45.358 INFO:teuthology.orchestra.run.smithi124.stdout: python3-rados-2:18.0.0-5706.ge7eb9853.el8.x86_64 2023-08-26T13:26:45.359 INFO:teuthology.orchestra.run.smithi124.stdout: python3-rbd-2:18.0.0-5706.ge7eb9853.el8.x86_64 2023-08-26T13:26:45.359 INFO:teuthology.orchestra.run.smithi124.stdout: python3-rgw-2:18.0.0-5706.ge7eb9853.el8.x86_64 2023-08-26T13:26:45.359 INFO:teuthology.orchestra.run.smithi124.stdout: thrift-0.13.0-2.el8.x86_64 2023-08-26T13:26:45.359 INFO:teuthology.orchestra.run.smithi124.stdout: 2023-08-26T13:26:45.359 INFO:teuthology.orchestra.run.smithi124.stdout:Complete! 2023-08-26T13:26:45.524 DEBUG:teuthology.orchestra.run.smithi124:> sudo yum -y install ceph-test 2023-08-26T13:26:45.871 INFO:teuthology.orchestra.run.smithi124.stdout:Updating Subscription Management repositories. 2023-08-26T13:26:47.523 INFO:teuthology.orchestra.run.smithi124.stdout:Last metadata expiration check: 0:01:04 ago on Sat 26 Aug 2023 01:25:43 PM UTC. 2023-08-26T13:26:48.173 INFO:teuthology.orchestra.run.smithi031.stdout:Installed products updated. 2023-08-26T13:26:48.461 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-08-26T13:26:48.462 INFO:teuthology.orchestra.run.smithi031.stdout:Upgraded: 2023-08-26T13:26:48.462 INFO:teuthology.orchestra.run.smithi031.stdout: librados2-2:18.0.0-5706.ge7eb9853.el8.x86_64 2023-08-26T13:26:48.462 INFO:teuthology.orchestra.run.smithi031.stdout: librbd1-2:18.0.0-5706.ge7eb9853.el8.x86_64 2023-08-26T13:26:48.462 INFO:teuthology.orchestra.run.smithi031.stdout:Installed: 2023-08-26T13:26:48.462 INFO:teuthology.orchestra.run.smithi031.stdout: ceph-base-2:18.0.0-5706.ge7eb9853.el8.x86_64 2023-08-26T13:26:48.463 INFO:teuthology.orchestra.run.smithi031.stdout: ceph-common-2:18.0.0-5706.ge7eb9853.el8.x86_64 2023-08-26T13:26:48.463 INFO:teuthology.orchestra.run.smithi031.stdout: ceph-radosgw-2:18.0.0-5706.ge7eb9853.el8.x86_64 2023-08-26T13:26:48.463 INFO:teuthology.orchestra.run.smithi031.stdout: ceph-selinux-2:18.0.0-5706.ge7eb9853.el8.x86_64 2023-08-26T13:26:48.463 INFO:teuthology.orchestra.run.smithi031.stdout: gperftools-libs-1:2.7-9.el8.x86_64 2023-08-26T13:26:48.463 INFO:teuthology.orchestra.run.smithi031.stdout: libcephfs2-2:18.0.0-5706.ge7eb9853.el8.x86_64 2023-08-26T13:26:48.464 INFO:teuthology.orchestra.run.smithi031.stdout: liboath-2.6.2-3.el8.x86_64 2023-08-26T13:26:48.464 INFO:teuthology.orchestra.run.smithi031.stdout: librabbitmq-0.9.0-3.el8.x86_64 2023-08-26T13:26:48.464 INFO:teuthology.orchestra.run.smithi031.stdout: libradosstriper1-2:18.0.0-5706.ge7eb9853.el8.x86_64 2023-08-26T13:26:48.464 INFO:teuthology.orchestra.run.smithi031.stdout: librdkafka-0.11.4-3.el8.x86_64 2023-08-26T13:26:48.464 INFO:teuthology.orchestra.run.smithi031.stdout: librgw2-2:18.0.0-5706.ge7eb9853.el8.x86_64 2023-08-26T13:26:48.465 INFO:teuthology.orchestra.run.smithi031.stdout: libunwind-1.3.1-3.el8.x86_64 2023-08-26T13:26:48.465 INFO:teuthology.orchestra.run.smithi031.stdout: lttng-ust-2.8.1-11.el8.x86_64 2023-08-26T13:26:48.465 INFO:teuthology.orchestra.run.smithi031.stdout: python3-ceph-argparse-2:18.0.0-5706.ge7eb9853.el8.x86_64 2023-08-26T13:26:48.465 INFO:teuthology.orchestra.run.smithi031.stdout: python3-ceph-common-2:18.0.0-5706.ge7eb9853.el8.x86_64 2023-08-26T13:26:48.465 INFO:teuthology.orchestra.run.smithi031.stdout: python3-cephfs-2:18.0.0-5706.ge7eb9853.el8.x86_64 2023-08-26T13:26:48.465 INFO:teuthology.orchestra.run.smithi031.stdout: python3-prettytable-0.7.2-14.el8.noarch 2023-08-26T13:26:48.466 INFO:teuthology.orchestra.run.smithi031.stdout: python3-rados-2:18.0.0-5706.ge7eb9853.el8.x86_64 2023-08-26T13:26:48.466 INFO:teuthology.orchestra.run.smithi031.stdout: python3-rbd-2:18.0.0-5706.ge7eb9853.el8.x86_64 2023-08-26T13:26:48.466 INFO:teuthology.orchestra.run.smithi031.stdout: python3-rgw-2:18.0.0-5706.ge7eb9853.el8.x86_64 2023-08-26T13:26:48.466 INFO:teuthology.orchestra.run.smithi031.stdout: thrift-0.13.0-2.el8.x86_64 2023-08-26T13:26:48.466 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-08-26T13:26:48.467 INFO:teuthology.orchestra.run.smithi031.stdout:Complete! 2023-08-26T13:26:48.736 DEBUG:teuthology.orchestra.run.smithi031:> sudo yum -y install ceph-test 2023-08-26T13:26:48.920 INFO:teuthology.orchestra.run.smithi124.stdout:Dependencies resolved. 2023-08-26T13:26:48.921 INFO:teuthology.orchestra.run.smithi124.stdout:============================================================================================ 2023-08-26T13:26:48.921 INFO:teuthology.orchestra.run.smithi124.stdout: Package Arch Version Repository Size 2023-08-26T13:26:48.921 INFO:teuthology.orchestra.run.smithi124.stdout:============================================================================================ 2023-08-26T13:26:48.922 INFO:teuthology.orchestra.run.smithi124.stdout:Installing: 2023-08-26T13:26:48.922 INFO:teuthology.orchestra.run.smithi124.stdout: ceph-test x86_64 2:18.0.0-5706.ge7eb9853.el8 ceph 48 M 2023-08-26T13:26:48.922 INFO:teuthology.orchestra.run.smithi124.stdout:Installing dependencies: 2023-08-26T13:26:48.922 INFO:teuthology.orchestra.run.smithi124.stdout: jq x86_64 1.6-3.el8 rhel-8-for-x86_64-appstream-rpms 202 k 2023-08-26T13:26:48.922 INFO:teuthology.orchestra.run.smithi124.stdout: libcephsqlite x86_64 2:18.0.0-5706.ge7eb9853.el8 ceph 139 k 2023-08-26T13:26:48.923 INFO:teuthology.orchestra.run.smithi124.stdout: oniguruma x86_64 6.8.2-2.el8 rhel-8-for-x86_64-appstream-rpms 187 k 2023-08-26T13:26:48.923 INFO:teuthology.orchestra.run.smithi124.stdout: socat x86_64 1.7.4.1-1.el8 rhel-8-for-x86_64-appstream-rpms 323 k 2023-08-26T13:26:48.923 INFO:teuthology.orchestra.run.smithi124.stdout: xmlstarlet x86_64 1.6.1-20.el8 lab-extras 70 k 2023-08-26T13:26:48.923 INFO:teuthology.orchestra.run.smithi124.stdout: 2023-08-26T13:26:48.923 INFO:teuthology.orchestra.run.smithi124.stdout:Transaction Summary 2023-08-26T13:26:48.924 INFO:teuthology.orchestra.run.smithi124.stdout:============================================================================================ 2023-08-26T13:26:48.924 INFO:teuthology.orchestra.run.smithi124.stdout:Install 6 Packages 2023-08-26T13:26:48.924 INFO:teuthology.orchestra.run.smithi124.stdout: 2023-08-26T13:26:48.925 INFO:teuthology.orchestra.run.smithi124.stdout:Total download size: 49 M 2023-08-26T13:26:48.925 INFO:teuthology.orchestra.run.smithi124.stdout:Installed size: 218 M 2023-08-26T13:26:48.926 INFO:teuthology.orchestra.run.smithi124.stdout:Downloading Packages: 2023-08-26T13:26:49.001 INFO:teuthology.orchestra.run.smithi124.stdout:(1/6): xmlstarlet-1.6.1-20.el8.x86_64.rpm 895 kB/s | 70 kB 00:00 2023-08-26T13:26:49.105 INFO:teuthology.orchestra.run.smithi031.stdout:Updating Subscription Management repositories. 2023-08-26T13:26:49.183 INFO:teuthology.orchestra.run.smithi124.stdout:(2/6): socat-1.7.4.1-1.el8.x86_64.rpm 1.8 MB/s | 323 kB 00:00 2023-08-26T13:26:49.358 INFO:teuthology.orchestra.run.smithi124.stdout:(3/6): oniguruma-6.8.2-2.el8.x86_64.rpm 1.0 MB/s | 187 kB 00:00 2023-08-26T13:26:49.467 INFO:teuthology.orchestra.run.smithi124.stdout:(4/6): jq-1.6-3.el8.x86_64.rpm 1.8 MB/s | 202 kB 00:00 2023-08-26T13:26:49.492 INFO:teuthology.orchestra.run.smithi124.stdout:(5/6): libcephsqlite-18.0.0-5706.ge7eb9853.el8. 244 kB/s | 139 kB 00:00 2023-08-26T13:26:50.682 INFO:teuthology.orchestra.run.smithi031.stdout:Last metadata expiration check: 0:01:07 ago on Sat 26 Aug 2023 01:25:43 PM UTC. 2023-08-26T13:26:52.187 INFO:teuthology.orchestra.run.smithi031.stdout:Dependencies resolved. 2023-08-26T13:26:52.188 INFO:teuthology.orchestra.run.smithi031.stdout:============================================================================================ 2023-08-26T13:26:52.188 INFO:teuthology.orchestra.run.smithi031.stdout: Package Arch Version Repository Size 2023-08-26T13:26:52.188 INFO:teuthology.orchestra.run.smithi031.stdout:============================================================================================ 2023-08-26T13:26:52.188 INFO:teuthology.orchestra.run.smithi031.stdout:Installing: 2023-08-26T13:26:52.189 INFO:teuthology.orchestra.run.smithi031.stdout: ceph-test x86_64 2:18.0.0-5706.ge7eb9853.el8 ceph 48 M 2023-08-26T13:26:52.189 INFO:teuthology.orchestra.run.smithi031.stdout:Installing dependencies: 2023-08-26T13:26:52.189 INFO:teuthology.orchestra.run.smithi031.stdout: jq x86_64 1.6-3.el8 rhel-8-for-x86_64-appstream-rpms 202 k 2023-08-26T13:26:52.189 INFO:teuthology.orchestra.run.smithi031.stdout: libcephsqlite x86_64 2:18.0.0-5706.ge7eb9853.el8 ceph 139 k 2023-08-26T13:26:52.189 INFO:teuthology.orchestra.run.smithi031.stdout: oniguruma x86_64 6.8.2-2.el8 rhel-8-for-x86_64-appstream-rpms 187 k 2023-08-26T13:26:52.190 INFO:teuthology.orchestra.run.smithi031.stdout: socat x86_64 1.7.4.1-1.el8 rhel-8-for-x86_64-appstream-rpms 323 k 2023-08-26T13:26:52.190 INFO:teuthology.orchestra.run.smithi031.stdout: xmlstarlet x86_64 1.6.1-20.el8 lab-extras 70 k 2023-08-26T13:26:52.190 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-08-26T13:26:52.190 INFO:teuthology.orchestra.run.smithi031.stdout:Transaction Summary 2023-08-26T13:26:52.190 INFO:teuthology.orchestra.run.smithi031.stdout:============================================================================================ 2023-08-26T13:26:52.191 INFO:teuthology.orchestra.run.smithi031.stdout:Install 6 Packages 2023-08-26T13:26:52.191 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-08-26T13:26:52.191 INFO:teuthology.orchestra.run.smithi031.stdout:Total download size: 49 M 2023-08-26T13:26:52.192 INFO:teuthology.orchestra.run.smithi031.stdout:Installed size: 218 M 2023-08-26T13:26:52.192 INFO:teuthology.orchestra.run.smithi031.stdout:Downloading Packages: 2023-08-26T13:26:52.231 INFO:teuthology.orchestra.run.smithi031.stdout:(1/6): xmlstarlet-1.6.1-20.el8.x86_64.rpm 1.7 MB/s | 70 kB 00:00 2023-08-26T13:26:52.339 INFO:teuthology.orchestra.run.smithi031.stdout:(2/6): socat-1.7.4.1-1.el8.x86_64.rpm 2.9 MB/s | 323 kB 00:00 2023-08-26T13:26:52.507 INFO:teuthology.orchestra.run.smithi124.stdout:(6/6): ceph-test-18.0.0-5706.ge7eb9853.el8.x86_ 13 MB/s | 48 MB 00:03 2023-08-26T13:26:52.509 INFO:teuthology.orchestra.run.smithi124.stdout:-------------------------------------------------------------------------------- 2023-08-26T13:26:52.510 INFO:teuthology.orchestra.run.smithi124.stdout:Total 14 MB/s | 49 MB 00:03 2023-08-26T13:26:52.531 INFO:teuthology.orchestra.run.smithi031.stdout:(3/6): oniguruma-6.8.2-2.el8.x86_64.rpm 980 kB/s | 187 kB 00:00 2023-08-26T13:26:52.546 INFO:teuthology.orchestra.run.smithi124.stdout:Running transaction check 2023-08-26T13:26:52.556 INFO:teuthology.orchestra.run.smithi031.stdout:(4/6): libcephsqlite-18.0.0-5706.ge7eb9853.el8. 380 kB/s | 139 kB 00:00 2023-08-26T13:26:52.567 INFO:teuthology.orchestra.run.smithi124.stdout:Transaction check succeeded. 2023-08-26T13:26:52.567 INFO:teuthology.orchestra.run.smithi124.stdout:Running transaction test 2023-08-26T13:26:52.698 INFO:teuthology.orchestra.run.smithi031.stdout:(5/6): jq-1.6-3.el8.x86_64.rpm 1.2 MB/s | 202 kB 00:00 2023-08-26T13:26:52.831 INFO:teuthology.orchestra.run.smithi124.stdout:Transaction test succeeded. 2023-08-26T13:26:52.835 INFO:teuthology.orchestra.run.smithi124.stdout:Running transaction 2023-08-26T13:26:53.276 INFO:teuthology.orchestra.run.smithi124.stdout: Preparing : 1/1 2023-08-26T13:26:53.409 INFO:teuthology.orchestra.run.smithi124.stdout: Installing : oniguruma-6.8.2-2.el8.x86_64 1/6 2023-08-26T13:26:53.496 INFO:teuthology.orchestra.run.smithi124.stdout: Running scriptlet: oniguruma-6.8.2-2.el8.x86_64 1/6 2023-08-26T13:26:53.728 INFO:teuthology.orchestra.run.smithi124.stdout: Installing : jq-1.6-3.el8.x86_64 2/6 2023-08-26T13:26:53.901 INFO:teuthology.orchestra.run.smithi124.stdout: Installing : socat-1.7.4.1-1.el8.x86_64 3/6 2023-08-26T13:26:54.045 INFO:teuthology.orchestra.run.smithi124.stdout: Installing : xmlstarlet-1.6.1-20.el8.x86_64 4/6 2023-08-26T13:26:54.258 INFO:teuthology.orchestra.run.smithi124.stdout: Installing : libcephsqlite-2:18.0.0-5706.ge7eb9853.el8.x86_64 5/6 2023-08-26T13:26:54.857 INFO:teuthology.orchestra.run.smithi031.stdout:(6/6): ceph-test-18.0.0-5706.ge7eb9853.el8.x86_ 18 MB/s | 48 MB 00:02 2023-08-26T13:26:54.859 INFO:teuthology.orchestra.run.smithi031.stdout:-------------------------------------------------------------------------------- 2023-08-26T13:26:54.859 INFO:teuthology.orchestra.run.smithi031.stdout:Total 18 MB/s | 49 MB 00:02 2023-08-26T13:26:54.898 INFO:teuthology.orchestra.run.smithi031.stdout:Running transaction check 2023-08-26T13:26:54.921 INFO:teuthology.orchestra.run.smithi031.stdout:Transaction check succeeded. 2023-08-26T13:26:54.921 INFO:teuthology.orchestra.run.smithi031.stdout:Running transaction test 2023-08-26T13:26:55.206 INFO:teuthology.orchestra.run.smithi031.stdout:Transaction test succeeded. 2023-08-26T13:26:55.210 INFO:teuthology.orchestra.run.smithi031.stdout:Running transaction 2023-08-26T13:26:55.670 INFO:teuthology.orchestra.run.smithi031.stdout: Preparing : 1/1 2023-08-26T13:26:55.792 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : oniguruma-6.8.2-2.el8.x86_64 1/6 2023-08-26T13:26:55.861 INFO:teuthology.orchestra.run.smithi031.stdout: Running scriptlet: oniguruma-6.8.2-2.el8.x86_64 1/6 2023-08-26T13:26:56.065 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : jq-1.6-3.el8.x86_64 2/6 2023-08-26T13:26:56.235 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : socat-1.7.4.1-1.el8.x86_64 3/6 2023-08-26T13:26:56.388 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : xmlstarlet-1.6.1-20.el8.x86_64 4/6 2023-08-26T13:26:56.584 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : libcephsqlite-2:18.0.0-5706.ge7eb9853.el8.x86_64 5/6 2023-08-26T13:26:58.892 INFO:teuthology.orchestra.run.smithi124.stdout: Running scriptlet: libcephsqlite-2:18.0.0-5706.ge7eb9853.el8.x86_64 5/6 2023-08-26T13:26:59.130 INFO:teuthology.orchestra.run.smithi124.stdout: Installing : ceph-test-2:18.0.0-5706.ge7eb9853.el8.x86_64 6/6 2023-08-26T13:26:59.359 INFO:teuthology.orchestra.run.smithi124.stdout: Running scriptlet: ceph-test-2:18.0.0-5706.ge7eb9853.el8.x86_64 6/6 2023-08-26T13:26:59.359 INFO:teuthology.orchestra.run.smithi124.stdout: Verifying : ceph-test-2:18.0.0-5706.ge7eb9853.el8.x86_64 1/6 2023-08-26T13:26:59.360 INFO:teuthology.orchestra.run.smithi124.stdout: Verifying : libcephsqlite-2:18.0.0-5706.ge7eb9853.el8.x86_64 2/6 2023-08-26T13:26:59.360 INFO:teuthology.orchestra.run.smithi124.stdout: Verifying : xmlstarlet-1.6.1-20.el8.x86_64 3/6 2023-08-26T13:26:59.360 INFO:teuthology.orchestra.run.smithi124.stdout: Verifying : socat-1.7.4.1-1.el8.x86_64 4/6 2023-08-26T13:26:59.360 INFO:teuthology.orchestra.run.smithi124.stdout: Verifying : oniguruma-6.8.2-2.el8.x86_64 5/6 2023-08-26T13:27:00.466 INFO:teuthology.orchestra.run.smithi124.stdout: Verifying : jq-1.6-3.el8.x86_64 6/6 2023-08-26T13:27:00.467 INFO:teuthology.orchestra.run.smithi124.stdout:Last metadata expiration check: 0:01:17 ago on Sat 26 Aug 2023 01:25:43 PM UTC. 2023-08-26T13:27:01.492 INFO:teuthology.orchestra.run.smithi031.stdout: Running scriptlet: libcephsqlite-2:18.0.0-5706.ge7eb9853.el8.x86_64 5/6 2023-08-26T13:27:01.764 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : ceph-test-2:18.0.0-5706.ge7eb9853.el8.x86_64 6/6 2023-08-26T13:27:01.995 INFO:teuthology.orchestra.run.smithi031.stdout: Running scriptlet: ceph-test-2:18.0.0-5706.ge7eb9853.el8.x86_64 6/6 2023-08-26T13:27:01.995 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : ceph-test-2:18.0.0-5706.ge7eb9853.el8.x86_64 1/6 2023-08-26T13:27:01.995 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : libcephsqlite-2:18.0.0-5706.ge7eb9853.el8.x86_64 2/6 2023-08-26T13:27:01.995 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : xmlstarlet-1.6.1-20.el8.x86_64 3/6 2023-08-26T13:27:01.996 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : socat-1.7.4.1-1.el8.x86_64 4/6 2023-08-26T13:27:01.996 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : oniguruma-6.8.2-2.el8.x86_64 5/6 2023-08-26T13:27:03.131 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : jq-1.6-3.el8.x86_64 6/6 2023-08-26T13:27:03.131 INFO:teuthology.orchestra.run.smithi031.stdout:Last metadata expiration check: 0:01:20 ago on Sat 26 Aug 2023 01:25:43 PM UTC. 2023-08-26T13:27:09.593 INFO:teuthology.orchestra.run.smithi031.stdout:Installed products updated. 2023-08-26T13:27:09.791 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-08-26T13:27:09.791 INFO:teuthology.orchestra.run.smithi031.stdout:Installed: 2023-08-26T13:27:09.792 INFO:teuthology.orchestra.run.smithi031.stdout: ceph-test-2:18.0.0-5706.ge7eb9853.el8.x86_64 2023-08-26T13:27:09.792 INFO:teuthology.orchestra.run.smithi031.stdout: jq-1.6-3.el8.x86_64 2023-08-26T13:27:09.792 INFO:teuthology.orchestra.run.smithi031.stdout: libcephsqlite-2:18.0.0-5706.ge7eb9853.el8.x86_64 2023-08-26T13:27:09.792 INFO:teuthology.orchestra.run.smithi031.stdout: oniguruma-6.8.2-2.el8.x86_64 2023-08-26T13:27:09.792 INFO:teuthology.orchestra.run.smithi031.stdout: socat-1.7.4.1-1.el8.x86_64 2023-08-26T13:27:09.793 INFO:teuthology.orchestra.run.smithi031.stdout: xmlstarlet-1.6.1-20.el8.x86_64 2023-08-26T13:27:09.793 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-08-26T13:27:09.793 INFO:teuthology.orchestra.run.smithi031.stdout:Complete! 2023-08-26T13:27:09.946 DEBUG:teuthology.orchestra.run.smithi031:> sudo yum -y install ceph 2023-08-26T13:27:09.984 INFO:teuthology.orchestra.run.smithi124.stdout:Installed products updated. 2023-08-26T13:27:10.165 INFO:teuthology.orchestra.run.smithi124.stdout: 2023-08-26T13:27:10.166 INFO:teuthology.orchestra.run.smithi124.stdout:Installed: 2023-08-26T13:27:10.166 INFO:teuthology.orchestra.run.smithi124.stdout: ceph-test-2:18.0.0-5706.ge7eb9853.el8.x86_64 2023-08-26T13:27:10.166 INFO:teuthology.orchestra.run.smithi124.stdout: jq-1.6-3.el8.x86_64 2023-08-26T13:27:10.166 INFO:teuthology.orchestra.run.smithi124.stdout: libcephsqlite-2:18.0.0-5706.ge7eb9853.el8.x86_64 2023-08-26T13:27:10.166 INFO:teuthology.orchestra.run.smithi124.stdout: oniguruma-6.8.2-2.el8.x86_64 2023-08-26T13:27:10.167 INFO:teuthology.orchestra.run.smithi124.stdout: socat-1.7.4.1-1.el8.x86_64 2023-08-26T13:27:10.167 INFO:teuthology.orchestra.run.smithi124.stdout: xmlstarlet-1.6.1-20.el8.x86_64 2023-08-26T13:27:10.167 INFO:teuthology.orchestra.run.smithi124.stdout: 2023-08-26T13:27:10.167 INFO:teuthology.orchestra.run.smithi124.stdout:Complete! 2023-08-26T13:27:10.290 DEBUG:teuthology.orchestra.run.smithi124:> sudo yum -y install ceph 2023-08-26T13:27:10.301 INFO:teuthology.orchestra.run.smithi031.stdout:Updating Subscription Management repositories. 2023-08-26T13:27:10.625 INFO:teuthology.orchestra.run.smithi124.stdout:Updating Subscription Management repositories. 2023-08-26T13:27:11.826 INFO:teuthology.orchestra.run.smithi031.stdout:Last metadata expiration check: 0:01:28 ago on Sat 26 Aug 2023 01:25:43 PM UTC. 2023-08-26T13:27:12.050 INFO:teuthology.orchestra.run.smithi124.stdout:Last metadata expiration check: 0:01:29 ago on Sat 26 Aug 2023 01:25:43 PM UTC. 2023-08-26T13:27:13.372 INFO:teuthology.orchestra.run.smithi031.stdout:Dependencies resolved. 2023-08-26T13:27:13.376 INFO:teuthology.orchestra.run.smithi031.stdout:======================================================================================================= 2023-08-26T13:27:13.376 INFO:teuthology.orchestra.run.smithi031.stdout: Package Arch Version Repository Size 2023-08-26T13:27:13.376 INFO:teuthology.orchestra.run.smithi031.stdout:======================================================================================================= 2023-08-26T13:27:13.377 INFO:teuthology.orchestra.run.smithi031.stdout:Installing: 2023-08-26T13:27:13.377 INFO:teuthology.orchestra.run.smithi031.stdout: ceph x86_64 2:18.0.0-5706.ge7eb9853.el8 ceph 6.4 k 2023-08-26T13:27:13.377 INFO:teuthology.orchestra.run.smithi031.stdout:Installing dependencies: 2023-08-26T13:27:13.377 INFO:teuthology.orchestra.run.smithi031.stdout: ceph-mds x86_64 2:18.0.0-5706.ge7eb9853.el8 ceph 2.3 M 2023-08-26T13:27:13.377 INFO:teuthology.orchestra.run.smithi031.stdout: ceph-mgr x86_64 2:18.0.0-5706.ge7eb9853.el8 ceph 1.6 M 2023-08-26T13:27:13.378 INFO:teuthology.orchestra.run.smithi031.stdout: ceph-mgr-modules-core noarch 2:18.0.0-5706.ge7eb9853.el8 ceph-noarch 242 k 2023-08-26T13:27:13.378 INFO:teuthology.orchestra.run.smithi031.stdout: ceph-mon x86_64 2:18.0.0-5706.ge7eb9853.el8 ceph 4.6 M 2023-08-26T13:27:13.378 INFO:teuthology.orchestra.run.smithi031.stdout: ceph-osd x86_64 2:18.0.0-5706.ge7eb9853.el8 ceph 18 M 2023-08-26T13:27:13.378 INFO:teuthology.orchestra.run.smithi031.stdout: python3-bcrypt x86_64 3.1.6-2.el8.1 epel 44 k 2023-08-26T13:27:13.379 INFO:teuthology.orchestra.run.smithi031.stdout: python3-beautifulsoup4 noarch 4.6.3-2.el8.1 epel 185 k 2023-08-26T13:27:13.379 INFO:teuthology.orchestra.run.smithi031.stdout: python3-cffi x86_64 1.11.5-5.el8 rhel-8-for-x86_64-baseos-rpms 238 k 2023-08-26T13:27:13.379 INFO:teuthology.orchestra.run.smithi031.stdout: python3-cheroot noarch 8.5.2-1.el8 epel 173 k 2023-08-26T13:27:13.379 INFO:teuthology.orchestra.run.smithi031.stdout: python3-cherrypy noarch 18.4.0-1.el8 epel 384 k 2023-08-26T13:27:13.380 INFO:teuthology.orchestra.run.smithi031.stdout: python3-cryptography x86_64 3.2.1-5.el8 rhel-8-for-x86_64-baseos-rpms 559 k 2023-08-26T13:27:13.380 INFO:teuthology.orchestra.run.smithi031.stdout: python3-jaraco noarch 6.2-6.el8 epel 11 k 2023-08-26T13:27:13.380 INFO:teuthology.orchestra.run.smithi031.stdout: python3-jaraco-functools noarch 2.0-4.el8 epel 18 k 2023-08-26T13:27:13.380 INFO:teuthology.orchestra.run.smithi031.stdout: python3-logutils noarch 0.3.5-11.el8 epel 49 k 2023-08-26T13:27:13.381 INFO:teuthology.orchestra.run.smithi031.stdout: python3-mako noarch 1.0.6-13.el8 rhel-8-for-x86_64-appstream-rpms 157 k 2023-08-26T13:27:13.381 INFO:teuthology.orchestra.run.smithi031.stdout: python3-markupsafe x86_64 0.23-19.el8 rhel-8-for-x86_64-appstream-rpms 39 k 2023-08-26T13:27:13.381 INFO:teuthology.orchestra.run.smithi031.stdout: python3-more-itertools noarch 7.2.0-3.el8 epel 59 k 2023-08-26T13:27:13.381 INFO:teuthology.orchestra.run.smithi031.stdout: python3-pecan noarch 1.3.2-9.el8 epel 283 k 2023-08-26T13:27:13.381 INFO:teuthology.orchestra.run.smithi031.stdout: python3-portend noarch 2.6-1.el8 epel 16 k 2023-08-26T13:27:13.382 INFO:teuthology.orchestra.run.smithi031.stdout: python3-pyOpenSSL noarch 19.0.0-1.el8 rhel-8-for-x86_64-appstream-rpms 103 k 2023-08-26T13:27:13.382 INFO:teuthology.orchestra.run.smithi031.stdout: python3-pycparser noarch 2.14-14.el8 rhel-8-for-x86_64-baseos-rpms 109 k 2023-08-26T13:27:13.382 INFO:teuthology.orchestra.run.smithi031.stdout: python3-pytz noarch 2017.2-9.el8 rhel-8-for-x86_64-appstream-rpms 54 k 2023-08-26T13:27:13.383 INFO:teuthology.orchestra.run.smithi031.stdout: python3-simplegeneric noarch 0.8.1-17.el8 epel 19 k 2023-08-26T13:27:13.383 INFO:teuthology.orchestra.run.smithi031.stdout: python3-singledispatch noarch 3.4.0.3-18.el8 epel 24 k 2023-08-26T13:27:13.383 INFO:teuthology.orchestra.run.smithi031.stdout: python3-tempora noarch 1.14.1-5.el8 epel 29 k 2023-08-26T13:27:13.383 INFO:teuthology.orchestra.run.smithi031.stdout: python3-trustme noarch 0.6.0-4.el8 epel 27 k 2023-08-26T13:27:13.384 INFO:teuthology.orchestra.run.smithi031.stdout: python3-waitress noarch 1.4.3-1.el8 epel 255 k 2023-08-26T13:27:13.384 INFO:teuthology.orchestra.run.smithi031.stdout: python3-webob noarch 1.8.5-1.el8.1 epel 251 k 2023-08-26T13:27:13.384 INFO:teuthology.orchestra.run.smithi031.stdout: python3-webtest noarch 2.0.33-1.el8 epel 85 k 2023-08-26T13:27:13.384 INFO:teuthology.orchestra.run.smithi031.stdout: python3-werkzeug noarch 0.12.2-4.el8 rhel-8-for-x86_64-appstream-rpms 457 k 2023-08-26T13:27:13.384 INFO:teuthology.orchestra.run.smithi031.stdout: python3-zc-lockfile noarch 2.0-2.el8 epel 23 k 2023-08-26T13:27:13.384 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-08-26T13:27:13.385 INFO:teuthology.orchestra.run.smithi031.stdout:Transaction Summary 2023-08-26T13:27:13.385 INFO:teuthology.orchestra.run.smithi031.stdout:======================================================================================================= 2023-08-26T13:27:13.385 INFO:teuthology.orchestra.run.smithi031.stdout:Install 32 Packages 2023-08-26T13:27:13.385 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-08-26T13:27:13.386 INFO:teuthology.orchestra.run.smithi031.stdout:Total download size: 30 M 2023-08-26T13:27:13.386 INFO:teuthology.orchestra.run.smithi031.stdout:Installed size: 120 M 2023-08-26T13:27:13.386 INFO:teuthology.orchestra.run.smithi031.stdout:Downloading Packages: 2023-08-26T13:27:13.426 INFO:teuthology.orchestra.run.smithi124.stdout:Dependencies resolved. 2023-08-26T13:27:13.429 INFO:teuthology.orchestra.run.smithi124.stdout:======================================================================================================= 2023-08-26T13:27:13.429 INFO:teuthology.orchestra.run.smithi124.stdout: Package Arch Version Repository Size 2023-08-26T13:27:13.429 INFO:teuthology.orchestra.run.smithi124.stdout:======================================================================================================= 2023-08-26T13:27:13.430 INFO:teuthology.orchestra.run.smithi124.stdout:Installing: 2023-08-26T13:27:13.430 INFO:teuthology.orchestra.run.smithi124.stdout: ceph x86_64 2:18.0.0-5706.ge7eb9853.el8 ceph 6.4 k 2023-08-26T13:27:13.430 INFO:teuthology.orchestra.run.smithi124.stdout:Installing dependencies: 2023-08-26T13:27:13.430 INFO:teuthology.orchestra.run.smithi124.stdout: ceph-mds x86_64 2:18.0.0-5706.ge7eb9853.el8 ceph 2.3 M 2023-08-26T13:27:13.430 INFO:teuthology.orchestra.run.smithi124.stdout: ceph-mgr x86_64 2:18.0.0-5706.ge7eb9853.el8 ceph 1.6 M 2023-08-26T13:27:13.431 INFO:teuthology.orchestra.run.smithi124.stdout: ceph-mgr-modules-core noarch 2:18.0.0-5706.ge7eb9853.el8 ceph-noarch 242 k 2023-08-26T13:27:13.431 INFO:teuthology.orchestra.run.smithi124.stdout: ceph-mon x86_64 2:18.0.0-5706.ge7eb9853.el8 ceph 4.6 M 2023-08-26T13:27:13.431 INFO:teuthology.orchestra.run.smithi124.stdout: ceph-osd x86_64 2:18.0.0-5706.ge7eb9853.el8 ceph 18 M 2023-08-26T13:27:13.431 INFO:teuthology.orchestra.run.smithi124.stdout: python3-bcrypt x86_64 3.1.6-2.el8.1 epel 44 k 2023-08-26T13:27:13.431 INFO:teuthology.orchestra.run.smithi124.stdout: python3-beautifulsoup4 noarch 4.6.3-2.el8.1 epel 185 k 2023-08-26T13:27:13.432 INFO:teuthology.orchestra.run.smithi124.stdout: python3-cffi x86_64 1.11.5-5.el8 rhel-8-for-x86_64-baseos-rpms 238 k 2023-08-26T13:27:13.432 INFO:teuthology.orchestra.run.smithi124.stdout: python3-cheroot noarch 8.5.2-1.el8 epel 173 k 2023-08-26T13:27:13.432 INFO:teuthology.orchestra.run.smithi124.stdout: python3-cherrypy noarch 18.4.0-1.el8 epel 384 k 2023-08-26T13:27:13.432 INFO:teuthology.orchestra.run.smithi124.stdout: python3-cryptography x86_64 3.2.1-5.el8 rhel-8-for-x86_64-baseos-rpms 559 k 2023-08-26T13:27:13.432 INFO:teuthology.orchestra.run.smithi124.stdout: python3-jaraco noarch 6.2-6.el8 epel 11 k 2023-08-26T13:27:13.433 INFO:teuthology.orchestra.run.smithi124.stdout: python3-jaraco-functools noarch 2.0-4.el8 epel 18 k 2023-08-26T13:27:13.433 INFO:teuthology.orchestra.run.smithi124.stdout: python3-logutils noarch 0.3.5-11.el8 epel 49 k 2023-08-26T13:27:13.433 INFO:teuthology.orchestra.run.smithi124.stdout: python3-mako noarch 1.0.6-13.el8 rhel-8-for-x86_64-appstream-rpms 157 k 2023-08-26T13:27:13.433 INFO:teuthology.orchestra.run.smithi124.stdout: python3-markupsafe x86_64 0.23-19.el8 rhel-8-for-x86_64-appstream-rpms 39 k 2023-08-26T13:27:13.433 INFO:teuthology.orchestra.run.smithi124.stdout: python3-more-itertools noarch 7.2.0-3.el8 epel 59 k 2023-08-26T13:27:13.434 INFO:teuthology.orchestra.run.smithi124.stdout: python3-pecan noarch 1.3.2-9.el8 epel 283 k 2023-08-26T13:27:13.434 INFO:teuthology.orchestra.run.smithi124.stdout: python3-portend noarch 2.6-1.el8 epel 16 k 2023-08-26T13:27:13.434 INFO:teuthology.orchestra.run.smithi124.stdout: python3-pyOpenSSL noarch 19.0.0-1.el8 rhel-8-for-x86_64-appstream-rpms 103 k 2023-08-26T13:27:13.434 INFO:teuthology.orchestra.run.smithi124.stdout: python3-pycparser noarch 2.14-14.el8 rhel-8-for-x86_64-baseos-rpms 109 k 2023-08-26T13:27:13.434 INFO:teuthology.orchestra.run.smithi124.stdout: python3-pytz noarch 2017.2-9.el8 rhel-8-for-x86_64-appstream-rpms 54 k 2023-08-26T13:27:13.434 INFO:teuthology.orchestra.run.smithi124.stdout: python3-simplegeneric noarch 0.8.1-17.el8 epel 19 k 2023-08-26T13:27:13.435 INFO:teuthology.orchestra.run.smithi124.stdout: python3-singledispatch noarch 3.4.0.3-18.el8 epel 24 k 2023-08-26T13:27:13.435 INFO:teuthology.orchestra.run.smithi124.stdout: python3-tempora noarch 1.14.1-5.el8 epel 29 k 2023-08-26T13:27:13.435 INFO:teuthology.orchestra.run.smithi124.stdout: python3-trustme noarch 0.6.0-4.el8 epel 27 k 2023-08-26T13:27:13.435 INFO:teuthology.orchestra.run.smithi124.stdout: python3-waitress noarch 1.4.3-1.el8 epel 255 k 2023-08-26T13:27:13.435 INFO:teuthology.orchestra.run.smithi124.stdout: python3-webob noarch 1.8.5-1.el8.1 epel 251 k 2023-08-26T13:27:13.436 INFO:teuthology.orchestra.run.smithi124.stdout: python3-webtest noarch 2.0.33-1.el8 epel 85 k 2023-08-26T13:27:13.436 INFO:teuthology.orchestra.run.smithi124.stdout: python3-werkzeug noarch 0.12.2-4.el8 rhel-8-for-x86_64-appstream-rpms 457 k 2023-08-26T13:27:13.436 INFO:teuthology.orchestra.run.smithi124.stdout: python3-zc-lockfile noarch 2.0-2.el8 epel 23 k 2023-08-26T13:27:13.436 INFO:teuthology.orchestra.run.smithi124.stdout: 2023-08-26T13:27:13.436 INFO:teuthology.orchestra.run.smithi124.stdout:Transaction Summary 2023-08-26T13:27:13.437 INFO:teuthology.orchestra.run.smithi124.stdout:======================================================================================================= 2023-08-26T13:27:13.437 INFO:teuthology.orchestra.run.smithi124.stdout:Install 32 Packages 2023-08-26T13:27:13.437 INFO:teuthology.orchestra.run.smithi124.stdout: 2023-08-26T13:27:13.438 INFO:teuthology.orchestra.run.smithi124.stdout:Total download size: 30 M 2023-08-26T13:27:13.438 INFO:teuthology.orchestra.run.smithi124.stdout:Installed size: 120 M 2023-08-26T13:27:13.438 INFO:teuthology.orchestra.run.smithi124.stdout:Downloading Packages: 2023-08-26T13:27:13.682 INFO:teuthology.orchestra.run.smithi031.stdout:(1/32): ceph-18.0.0-5706.ge7eb9853.el8.x86_64.r 24 kB/s | 6.4 kB 00:00 2023-08-26T13:27:13.724 INFO:teuthology.orchestra.run.smithi031.stdout:(2/32): ceph-mds-18.0.0-5706.ge7eb9853.el8.x86_ 7.5 MB/s | 2.3 MB 00:00 2023-08-26T13:27:14.367 INFO:teuthology.orchestra.run.smithi124.stdout:(1/32): ceph-18.0.0-5706.ge7eb9853.el8.x86_64.r 14 kB/s | 6.4 kB 00:00 2023-08-26T13:27:14.591 INFO:teuthology.orchestra.run.smithi031.stdout:(3/32): ceph-osd-18.0.0-5706.ge7eb9853.el8.x86_ 21 MB/s | 18 MB 00:00 2023-08-26T13:27:14.641 INFO:teuthology.orchestra.run.smithi031.stdout:(4/32): ceph-mgr-modules-core-18.0.0-5706.ge7eb 4.8 MB/s | 242 kB 00:00 2023-08-26T13:27:14.741 INFO:teuthology.orchestra.run.smithi031.stdout:(5/32): ceph-mgr-18.0.0-5706.ge7eb9853.el8.x86_ 1.2 MB/s | 1.6 MB 00:01 2023-08-26T13:27:14.792 INFO:teuthology.orchestra.run.smithi124.stdout:(2/32): ceph-mgr-18.0.0-5706.ge7eb9853.el8.x86_ 1.8 MB/s | 1.6 MB 00:00 2023-08-26T13:27:14.984 INFO:teuthology.orchestra.run.smithi124.stdout:(3/32): ceph-mds-18.0.0-5706.ge7eb9853.el8.x86_ 2.1 MB/s | 2.3 MB 00:01 2023-08-26T13:27:15.059 INFO:teuthology.orchestra.run.smithi124.stdout:(4/32): ceph-mon-18.0.0-5706.ge7eb9853.el8.x86_ 6.7 MB/s | 4.6 MB 00:00 2023-08-26T13:27:15.083 INFO:teuthology.orchestra.run.smithi031.stdout:(6/32): python3-bcrypt-3.1.6-2.el8.1.x86_64.rpm 99 kB/s | 44 kB 00:00 2023-08-26T13:27:15.118 INFO:teuthology.orchestra.run.smithi124.stdout:(5/32): ceph-mgr-modules-core-18.0.0-5706.ge7eb 1.8 MB/s | 242 kB 00:00 2023-08-26T13:27:15.292 INFO:teuthology.orchestra.run.smithi031.stdout:(7/32): python3-beautifulsoup4-4.6.3-2.el8.1.no 337 kB/s | 185 kB 00:00 2023-08-26T13:27:15.393 INFO:teuthology.orchestra.run.smithi124.stdout:(6/32): python3-bcrypt-3.1.6-2.el8.1.x86_64.rpm 131 kB/s | 44 kB 00:00 2023-08-26T13:27:15.417 INFO:teuthology.orchestra.run.smithi031.stdout:(8/32): python3-cheroot-8.5.2-1.el8.noarch.rpm 517 kB/s | 173 kB 00:00 2023-08-26T13:27:15.526 INFO:teuthology.orchestra.run.smithi031.stdout:(9/32): python3-jaraco-6.2-6.el8.noarch.rpm 102 kB/s | 11 kB 00:00 2023-08-26T13:27:15.527 INFO:teuthology.orchestra.run.smithi124.stdout:(7/32): python3-beautifulsoup4-4.6.3-2.el8.1.no 453 kB/s | 185 kB 00:00 2023-08-26T13:27:15.611 INFO:teuthology.orchestra.run.smithi124.stdout:(8/32): python3-cheroot-8.5.2-1.el8.noarch.rpm 795 kB/s | 173 kB 00:00 2023-08-26T13:27:15.642 INFO:teuthology.orchestra.run.smithi031.stdout:(10/32): python3-cherrypy-18.4.0-1.el8.noarch.r 1.1 MB/s | 384 kB 00:00 2023-08-26T13:27:15.668 INFO:teuthology.orchestra.run.smithi031.stdout:(11/32): python3-jaraco-functools-2.0-4.el8.noa 124 kB/s | 18 kB 00:00 2023-08-26T13:27:15.719 INFO:teuthology.orchestra.run.smithi124.stdout:(9/32): python3-cherrypy-18.4.0-1.el8.noarch.rp 2.0 MB/s | 384 kB 00:00 2023-08-26T13:27:15.736 INFO:teuthology.orchestra.run.smithi124.stdout:(10/32): python3-jaraco-6.2-6.el8.noarch.rpm 89 kB/s | 11 kB 00:00 2023-08-26T13:27:15.768 INFO:teuthology.orchestra.run.smithi031.stdout:(12/32): python3-logutils-0.3.5-11.el8.noarch.r 389 kB/s | 49 kB 00:00 2023-08-26T13:27:15.785 INFO:teuthology.orchestra.run.smithi031.stdout:(13/32): python3-more-itertools-7.2.0-3.el8.noa 502 kB/s | 59 kB 00:00 2023-08-26T13:27:15.845 INFO:teuthology.orchestra.run.smithi124.stdout:(11/32): python3-jaraco-functools-2.0-4.el8.noa 140 kB/s | 18 kB 00:00 2023-08-26T13:27:15.862 INFO:teuthology.orchestra.run.smithi124.stdout:(12/32): python3-logutils-0.3.5-11.el8.noarch.r 388 kB/s | 49 kB 00:00 2023-08-26T13:27:15.918 INFO:teuthology.orchestra.run.smithi031.stdout:(14/32): python3-portend-2.6-1.el8.noarch.rpm 117 kB/s | 16 kB 00:00 2023-08-26T13:27:15.954 INFO:teuthology.orchestra.run.smithi124.stdout:(13/32): python3-more-itertools-7.2.0-3.el8.noa 541 kB/s | 59 kB 00:00 2023-08-26T13:27:16.002 INFO:teuthology.orchestra.run.smithi031.stdout:(15/32): python3-pecan-1.3.2-9.el8.noarch.rpm 1.2 MB/s | 283 kB 00:00 2023-08-26T13:27:16.069 INFO:teuthology.orchestra.run.smithi031.stdout:(16/32): ceph-mon-18.0.0-5706.ge7eb9853.el8.x86 1.9 MB/s | 4.6 MB 00:02 2023-08-26T13:27:16.086 INFO:teuthology.orchestra.run.smithi031.stdout:(17/32): python3-simplegeneric-0.8.1-17.el8.noa 112 kB/s | 19 kB 00:00 2023-08-26T13:27:16.119 INFO:teuthology.orchestra.run.smithi031.stdout:(18/32): python3-singledispatch-3.4.0.3-18.el8. 204 kB/s | 24 kB 00:00 2023-08-26T13:27:16.154 INFO:teuthology.orchestra.run.smithi124.stdout:(14/32): ceph-osd-18.0.0-5706.ge7eb9853.el8.x86 13 MB/s | 18 MB 00:01 2023-08-26T13:27:16.180 INFO:teuthology.orchestra.run.smithi124.stdout:(15/32): python3-portend-2.6-1.el8.noarch.rpm 69 kB/s | 16 kB 00:00 2023-08-26T13:27:16.211 INFO:teuthology.orchestra.run.smithi031.stdout:(19/32): python3-trustme-0.6.0-4.el8.noarch.rpm 220 kB/s | 27 kB 00:00 2023-08-26T13:27:16.213 INFO:teuthology.orchestra.run.smithi124.stdout:(16/32): python3-pecan-1.3.2-9.el8.noarch.rpm 806 kB/s | 283 kB 00:00 2023-08-26T13:27:16.264 INFO:teuthology.orchestra.run.smithi124.stdout:(17/32): python3-simplegeneric-0.8.1-17.el8.noa 172 kB/s | 19 kB 00:00 2023-08-26T13:27:16.331 INFO:teuthology.orchestra.run.smithi124.stdout:(18/32): python3-tempora-1.14.1-5.el8.noarch.rp 250 kB/s | 29 kB 00:00 2023-08-26T13:27:16.362 INFO:teuthology.orchestra.run.smithi031.stdout:(20/32): python3-waitress-1.4.3-1.el8.noarch.rp 1.0 MB/s | 255 kB 00:00 2023-08-26T13:27:16.395 INFO:teuthology.orchestra.run.smithi031.stdout:(21/32): python3-tempora-1.14.1-5.el8.noarch.rp 90 kB/s | 29 kB 00:00 2023-08-26T13:27:16.406 INFO:teuthology.orchestra.run.smithi124.stdout:(19/32): python3-trustme-0.6.0-4.el8.noarch.rpm 194 kB/s | 27 kB 00:00 2023-08-26T13:27:16.437 INFO:teuthology.orchestra.run.smithi031.stdout:(22/32): python3-webob-1.8.5-1.el8.1.noarch.rpm 1.1 MB/s | 251 kB 00:00 2023-08-26T13:27:16.448 INFO:teuthology.orchestra.run.smithi124.stdout:(20/32): python3-singledispatch-3.4.0.3-18.el8. 89 kB/s | 24 kB 00:00 2023-08-26T13:27:16.473 INFO:teuthology.orchestra.run.smithi124.stdout:(21/32): python3-waitress-1.4.3-1.el8.noarch.rp 1.7 MB/s | 255 kB 00:00 2023-08-26T13:27:16.479 INFO:teuthology.orchestra.run.smithi031.stdout:(23/32): python3-webtest-2.0.33-1.el8.noarch.rp 730 kB/s | 85 kB 00:00 2023-08-26T13:27:16.538 INFO:teuthology.orchestra.run.smithi031.stdout:(24/32): python3-zc-lockfile-2.0-2.el8.noarch.r 161 kB/s | 23 kB 00:00 2023-08-26T13:27:16.597 INFO:teuthology.orchestra.run.smithi031.stdout:(25/32): python3-pycparser-2.14-14.el8.noarch.r 686 kB/s | 109 kB 00:00 2023-08-26T13:27:16.599 INFO:teuthology.orchestra.run.smithi124.stdout:(22/32): python3-zc-lockfile-2.0-2.el8.noarch.r 183 kB/s | 23 kB 00:00 2023-08-26T13:27:16.624 INFO:teuthology.orchestra.run.smithi124.stdout:(23/32): python3-webob-1.8.5-1.el8.1.noarch.rpm 1.1 MB/s | 251 kB 00:00 2023-08-26T13:27:16.647 INFO:teuthology.orchestra.run.smithi031.stdout:(26/32): python3-cryptography-3.2.1-5.el8.x86_6 3.3 MB/s | 559 kB 00:00 2023-08-26T13:27:16.650 INFO:teuthology.orchestra.run.smithi124.stdout:(24/32): python3-webtest-2.0.33-1.el8.noarch.rp 424 kB/s | 85 kB 00:00 2023-08-26T13:27:16.722 INFO:teuthology.orchestra.run.smithi031.stdout:(27/32): python3-cffi-1.11.5-5.el8.x86_64.rpm 1.3 MB/s | 238 kB 00:00 2023-08-26T13:27:16.747 INFO:teuthology.orchestra.run.smithi031.stdout:(28/32): python3-pytz-2017.2-9.el8.noarch.rpm 360 kB/s | 54 kB 00:00 2023-08-26T13:27:16.766 INFO:teuthology.orchestra.run.smithi124.stdout:(25/32): python3-pycparser-2.14-14.el8.noarch.r 653 kB/s | 109 kB 00:00 2023-08-26T13:27:16.789 INFO:teuthology.orchestra.run.smithi031.stdout:(29/32): python3-markupsafe-0.23-19.el8.x86_64. 273 kB/s | 39 kB 00:00 2023-08-26T13:27:16.842 INFO:teuthology.orchestra.run.smithi124.stdout:(26/32): python3-cffi-1.11.5-5.el8.x86_64.rpm 1.2 MB/s | 238 kB 00:00 2023-08-26T13:27:16.876 INFO:teuthology.orchestra.run.smithi124.stdout:(27/32): python3-cryptography-3.2.1-5.el8.x86_6 2.4 MB/s | 559 kB 00:00 2023-08-26T13:27:16.881 INFO:teuthology.orchestra.run.smithi031.stdout:(30/32): python3-mako-1.0.6-13.el8.noarch.rpm 989 kB/s | 157 kB 00:00 2023-08-26T13:27:16.926 INFO:teuthology.orchestra.run.smithi124.stdout:(28/32): python3-pytz-2017.2-9.el8.noarch.rpm 341 kB/s | 54 kB 00:00 2023-08-26T13:27:16.940 INFO:teuthology.orchestra.run.smithi031.stdout:(31/32): python3-pyOpenSSL-19.0.0-1.el8.noarch. 690 kB/s | 103 kB 00:00 2023-08-26T13:27:16.951 INFO:teuthology.orchestra.run.smithi124.stdout:(29/32): python3-markupsafe-0.23-19.el8.x86_64. 357 kB/s | 39 kB 00:00 2023-08-26T13:27:16.973 INFO:teuthology.orchestra.run.smithi031.stdout:(32/32): python3-werkzeug-0.12.2-4.el8.noarch.r 2.0 MB/s | 457 kB 00:00 2023-08-26T13:27:16.975 INFO:teuthology.orchestra.run.smithi031.stdout:-------------------------------------------------------------------------------- 2023-08-26T13:27:16.975 INFO:teuthology.orchestra.run.smithi031.stdout:Total 8.4 MB/s | 30 MB 00:03 2023-08-26T13:27:17.010 INFO:teuthology.orchestra.run.smithi124.stdout:(30/32): python3-mako-1.0.6-13.el8.noarch.rpm 1.1 MB/s | 157 kB 00:00 2023-08-26T13:27:17.068 INFO:teuthology.orchestra.run.smithi031.stdout:Running transaction check 2023-08-26T13:27:17.069 INFO:teuthology.orchestra.run.smithi124.stdout:(31/32): python3-werkzeug-0.12.2-4.el8.noarch.r 3.1 MB/s | 457 kB 00:00 2023-08-26T13:27:17.108 INFO:teuthology.orchestra.run.smithi031.stdout:Transaction check succeeded. 2023-08-26T13:27:17.109 INFO:teuthology.orchestra.run.smithi031.stdout:Running transaction test 2023-08-26T13:27:17.110 INFO:teuthology.orchestra.run.smithi124.stdout:(32/32): python3-pyOpenSSL-19.0.0-1.el8.noarch. 652 kB/s | 103 kB 00:00 2023-08-26T13:27:17.112 INFO:teuthology.orchestra.run.smithi124.stdout:-------------------------------------------------------------------------------- 2023-08-26T13:27:17.112 INFO:teuthology.orchestra.run.smithi124.stdout:Total 8.2 MB/s | 30 MB 00:03 2023-08-26T13:27:17.201 INFO:teuthology.orchestra.run.smithi124.stdout:Running transaction check 2023-08-26T13:27:17.237 INFO:teuthology.orchestra.run.smithi124.stdout:Transaction check succeeded. 2023-08-26T13:27:17.238 INFO:teuthology.orchestra.run.smithi124.stdout:Running transaction test 2023-08-26T13:27:17.334 INFO:teuthology.orchestra.run.smithi031.stdout:Transaction test succeeded. 2023-08-26T13:27:17.338 INFO:teuthology.orchestra.run.smithi031.stdout:Running transaction 2023-08-26T13:27:17.447 INFO:teuthology.orchestra.run.smithi124.stdout:Transaction test succeeded. 2023-08-26T13:27:17.451 INFO:teuthology.orchestra.run.smithi124.stdout:Running transaction 2023-08-26T13:27:18.429 INFO:teuthology.orchestra.run.smithi031.stdout: Preparing : 1/1 2023-08-26T13:27:18.557 INFO:teuthology.orchestra.run.smithi124.stdout: Preparing : 1/1 2023-08-26T13:27:18.648 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : python3-more-itertools-7.2.0-3.el8.noarch 1/32 2023-08-26T13:27:18.769 INFO:teuthology.orchestra.run.smithi124.stdout: Installing : python3-more-itertools-7.2.0-3.el8.noarch 1/32 2023-08-26T13:27:18.928 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : python3-webob-1.8.5-1.el8.1.noarch 2/32 2023-08-26T13:27:19.055 INFO:teuthology.orchestra.run.smithi124.stdout: Installing : python3-webob-1.8.5-1.el8.1.noarch 2/32 2023-08-26T13:27:19.118 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : python3-werkzeug-0.12.2-4.el8.noarch 3/32 2023-08-26T13:27:19.275 INFO:teuthology.orchestra.run.smithi124.stdout: Installing : python3-werkzeug-0.12.2-4.el8.noarch 3/32 2023-08-26T13:27:19.353 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : python3-markupsafe-0.23-19.el8.x86_64 4/32 2023-08-26T13:27:19.505 INFO:teuthology.orchestra.run.smithi124.stdout: Installing : python3-markupsafe-0.23-19.el8.x86_64 4/32 2023-08-26T13:27:19.561 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : python3-mako-1.0.6-13.el8.noarch 5/32 2023-08-26T13:27:19.729 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : python3-pytz-2017.2-9.el8.noarch 6/32 2023-08-26T13:27:19.734 INFO:teuthology.orchestra.run.smithi124.stdout: Installing : python3-mako-1.0.6-13.el8.noarch 5/32 2023-08-26T13:27:19.892 INFO:teuthology.orchestra.run.smithi124.stdout: Installing : python3-pytz-2017.2-9.el8.noarch 6/32 2023-08-26T13:27:19.953 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : python3-pycparser-2.14-14.el8.noarch 7/32 2023-08-26T13:27:20.107 INFO:teuthology.orchestra.run.smithi124.stdout: Installing : python3-pycparser-2.14-14.el8.noarch 7/32 2023-08-26T13:27:20.343 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : python3-cffi-1.11.5-5.el8.x86_64 8/32 2023-08-26T13:27:20.502 INFO:teuthology.orchestra.run.smithi124.stdout: Installing : python3-cffi-1.11.5-5.el8.x86_64 8/32 2023-08-26T13:27:20.619 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : python3-cryptography-3.2.1-5.el8.x86_64 9/32 2023-08-26T13:27:20.783 INFO:teuthology.orchestra.run.smithi124.stdout: Installing : python3-cryptography-3.2.1-5.el8.x86_64 9/32 2023-08-26T13:27:20.788 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : python3-pyOpenSSL-19.0.0-1.el8.noarch 10/32 2023-08-26T13:27:20.965 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : python3-trustme-0.6.0-4.el8.noarch 11/32 2023-08-26T13:27:20.987 INFO:teuthology.orchestra.run.smithi124.stdout: Installing : python3-pyOpenSSL-19.0.0-1.el8.noarch 10/32 2023-08-26T13:27:21.147 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : python3-bcrypt-3.1.6-2.el8.1.x86_64 12/32 2023-08-26T13:27:21.164 INFO:teuthology.orchestra.run.smithi124.stdout: Installing : python3-trustme-0.6.0-4.el8.noarch 11/32 2023-08-26T13:27:21.364 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : python3-zc-lockfile-2.0-2.el8.noarch 13/32 2023-08-26T13:27:21.379 INFO:teuthology.orchestra.run.smithi124.stdout: Installing : python3-bcrypt-3.1.6-2.el8.1.x86_64 12/32 2023-08-26T13:27:21.512 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : python3-waitress-1.4.3-1.el8.noarch 14/32 2023-08-26T13:27:21.622 INFO:teuthology.orchestra.run.smithi124.stdout: Installing : python3-zc-lockfile-2.0-2.el8.noarch 13/32 2023-08-26T13:27:21.667 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : python3-singledispatch-3.4.0.3-18.el8.noarch 15/32 2023-08-26T13:27:21.786 INFO:teuthology.orchestra.run.smithi124.stdout: Installing : python3-waitress-1.4.3-1.el8.noarch 14/32 2023-08-26T13:27:21.799 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : python3-simplegeneric-0.8.1-17.el8.noarch 16/32 2023-08-26T13:27:21.958 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : python3-logutils-0.3.5-11.el8.noarch 17/32 2023-08-26T13:27:21.959 INFO:teuthology.orchestra.run.smithi124.stdout: Installing : python3-singledispatch-3.4.0.3-18.el8.noarch 15/32 2023-08-26T13:27:22.086 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : python3-jaraco-6.2-6.el8.noarch 18/32 2023-08-26T13:27:22.122 INFO:teuthology.orchestra.run.smithi124.stdout: Installing : python3-simplegeneric-0.8.1-17.el8.noarch 16/32 2023-08-26T13:27:22.282 INFO:teuthology.orchestra.run.smithi124.stdout: Installing : python3-logutils-0.3.5-11.el8.noarch 17/32 2023-08-26T13:27:22.338 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : python3-jaraco-functools-2.0-4.el8.noarch 19/32 2023-08-26T13:27:22.402 INFO:teuthology.orchestra.run.smithi124.stdout: Installing : python3-jaraco-6.2-6.el8.noarch 18/32 2023-08-26T13:27:22.526 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : python3-cheroot-8.5.2-1.el8.noarch 20/32 2023-08-26T13:27:22.648 INFO:teuthology.orchestra.run.smithi124.stdout: Installing : python3-jaraco-functools-2.0-4.el8.noarch 19/32 2023-08-26T13:27:22.702 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : python3-tempora-1.14.1-5.el8.noarch 21/32 2023-08-26T13:27:22.866 INFO:teuthology.orchestra.run.smithi124.stdout: Installing : python3-cheroot-8.5.2-1.el8.noarch 20/32 2023-08-26T13:27:22.956 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : python3-portend-2.6-1.el8.noarch 22/32 2023-08-26T13:27:23.035 INFO:teuthology.orchestra.run.smithi124.stdout: Installing : python3-tempora-1.14.1-5.el8.noarch 21/32 2023-08-26T13:27:23.188 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : python3-cherrypy-18.4.0-1.el8.noarch 23/32 2023-08-26T13:27:23.308 INFO:teuthology.orchestra.run.smithi124.stdout: Installing : python3-portend-2.6-1.el8.noarch 22/32 2023-08-26T13:27:23.408 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : python3-beautifulsoup4-4.6.3-2.el8.1.noarch 24/32 2023-08-26T13:27:23.550 INFO:teuthology.orchestra.run.smithi124.stdout: Installing : python3-cherrypy-18.4.0-1.el8.noarch 23/32 2023-08-26T13:27:23.740 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : python3-webtest-2.0.33-1.el8.noarch 25/32 2023-08-26T13:27:23.788 INFO:teuthology.orchestra.run.smithi124.stdout: Installing : python3-beautifulsoup4-4.6.3-2.el8.1.noarch 24/32 2023-08-26T13:27:24.052 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : python3-pecan-1.3.2-9.el8.noarch 26/32 2023-08-26T13:27:24.126 INFO:teuthology.orchestra.run.smithi124.stdout: Installing : python3-webtest-2.0.33-1.el8.noarch 25/32 2023-08-26T13:27:24.366 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : ceph-mgr-modules-core-2:18.0.0-5706.ge7eb9853.el8. 27/32 2023-08-26T13:27:24.485 INFO:teuthology.orchestra.run.smithi124.stdout: Installing : python3-pecan-1.3.2-9.el8.noarch 26/32 2023-08-26T13:27:24.512 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : ceph-mgr-2:18.0.0-5706.ge7eb9853.el8.x86_64 28/32 2023-08-26T13:27:24.793 INFO:teuthology.orchestra.run.smithi124.stdout: Installing : ceph-mgr-modules-core-2:18.0.0-5706.ge7eb9853.el8. 27/32 2023-08-26T13:27:24.961 INFO:teuthology.orchestra.run.smithi124.stdout: Installing : ceph-mgr-2:18.0.0-5706.ge7eb9853.el8.x86_64 28/32 2023-08-26T13:27:26.199 INFO:teuthology.orchestra.run.smithi031.stdout: Running scriptlet: ceph-mgr-2:18.0.0-5706.ge7eb9853.el8.x86_64 28/32 2023-08-26T13:27:26.379 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : ceph-osd-2:18.0.0-5706.ge7eb9853.el8.x86_64 29/32 2023-08-26T13:27:26.545 INFO:teuthology.orchestra.run.smithi124.stdout: Running scriptlet: ceph-mgr-2:18.0.0-5706.ge7eb9853.el8.x86_64 28/32 2023-08-26T13:27:26.769 INFO:teuthology.orchestra.run.smithi124.stdout: Installing : ceph-osd-2:18.0.0-5706.ge7eb9853.el8.x86_64 29/32 2023-08-26T13:27:26.867 INFO:teuthology.orchestra.run.smithi031.stdout: Running scriptlet: ceph-osd-2:18.0.0-5706.ge7eb9853.el8.x86_64 29/32 2023-08-26T13:27:27.013 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : ceph-mon-2:18.0.0-5706.ge7eb9853.el8.x86_64 30/32 2023-08-26T13:27:27.228 INFO:teuthology.orchestra.run.smithi124.stdout: Running scriptlet: ceph-osd-2:18.0.0-5706.ge7eb9853.el8.x86_64 29/32 2023-08-26T13:27:27.254 INFO:teuthology.orchestra.run.smithi031.stdout: Running scriptlet: ceph-mon-2:18.0.0-5706.ge7eb9853.el8.x86_64 30/32 2023-08-26T13:27:27.405 INFO:teuthology.orchestra.run.smithi124.stdout: Installing : ceph-mon-2:18.0.0-5706.ge7eb9853.el8.x86_64 30/32 2023-08-26T13:27:27.454 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : ceph-mds-2:18.0.0-5706.ge7eb9853.el8.x86_64 31/32 2023-08-26T13:27:27.473 INFO:teuthology.orchestra.run.smithi031.stdout: Running scriptlet: ceph-mds-2:18.0.0-5706.ge7eb9853.el8.x86_64 31/32 2023-08-26T13:27:27.561 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : ceph-2:18.0.0-5706.ge7eb9853.el8.x86_64 32/32 2023-08-26T13:27:27.630 INFO:teuthology.orchestra.run.smithi124.stdout: Running scriptlet: ceph-mon-2:18.0.0-5706.ge7eb9853.el8.x86_64 30/32 2023-08-26T13:27:27.804 INFO:teuthology.orchestra.run.smithi124.stdout: Installing : ceph-mds-2:18.0.0-5706.ge7eb9853.el8.x86_64 31/32 2023-08-26T13:27:27.822 INFO:teuthology.orchestra.run.smithi124.stdout: Running scriptlet: ceph-mds-2:18.0.0-5706.ge7eb9853.el8.x86_64 31/32 2023-08-26T13:27:27.926 INFO:teuthology.orchestra.run.smithi031.stdout: Running scriptlet: ceph-2:18.0.0-5706.ge7eb9853.el8.x86_64 32/32 2023-08-26T13:27:27.926 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : ceph-2:18.0.0-5706.ge7eb9853.el8.x86_64 1/32 2023-08-26T13:27:27.926 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : ceph-mds-2:18.0.0-5706.ge7eb9853.el8.x86_64 2/32 2023-08-26T13:27:27.926 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : ceph-mgr-2:18.0.0-5706.ge7eb9853.el8.x86_64 3/32 2023-08-26T13:27:27.926 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : ceph-mon-2:18.0.0-5706.ge7eb9853.el8.x86_64 4/32 2023-08-26T13:27:27.927 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : ceph-osd-2:18.0.0-5706.ge7eb9853.el8.x86_64 5/32 2023-08-26T13:27:27.927 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : ceph-mgr-modules-core-2:18.0.0-5706.ge7eb9853.el8. 6/32 2023-08-26T13:27:27.927 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : python3-bcrypt-3.1.6-2.el8.1.x86_64 7/32 2023-08-26T13:27:27.927 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : python3-beautifulsoup4-4.6.3-2.el8.1.noarch 8/32 2023-08-26T13:27:27.928 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : python3-cheroot-8.5.2-1.el8.noarch 9/32 2023-08-26T13:27:27.928 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : python3-cherrypy-18.4.0-1.el8.noarch 10/32 2023-08-26T13:27:27.928 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : python3-jaraco-6.2-6.el8.noarch 11/32 2023-08-26T13:27:27.928 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : python3-jaraco-functools-2.0-4.el8.noarch 12/32 2023-08-26T13:27:27.928 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : python3-logutils-0.3.5-11.el8.noarch 13/32 2023-08-26T13:27:27.929 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : python3-more-itertools-7.2.0-3.el8.noarch 14/32 2023-08-26T13:27:27.929 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : python3-pecan-1.3.2-9.el8.noarch 15/32 2023-08-26T13:27:27.929 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : python3-portend-2.6-1.el8.noarch 16/32 2023-08-26T13:27:27.929 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : python3-simplegeneric-0.8.1-17.el8.noarch 17/32 2023-08-26T13:27:27.929 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : python3-singledispatch-3.4.0.3-18.el8.noarch 18/32 2023-08-26T13:27:27.929 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : python3-tempora-1.14.1-5.el8.noarch 19/32 2023-08-26T13:27:27.930 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : python3-trustme-0.6.0-4.el8.noarch 20/32 2023-08-26T13:27:27.930 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : python3-waitress-1.4.3-1.el8.noarch 21/32 2023-08-26T13:27:27.930 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : python3-webob-1.8.5-1.el8.1.noarch 22/32 2023-08-26T13:27:27.930 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : python3-webtest-2.0.33-1.el8.noarch 23/32 2023-08-26T13:27:27.930 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : python3-zc-lockfile-2.0-2.el8.noarch 24/32 2023-08-26T13:27:27.931 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : python3-pycparser-2.14-14.el8.noarch 25/32 2023-08-26T13:27:27.931 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : python3-cryptography-3.2.1-5.el8.x86_64 26/32 2023-08-26T13:27:27.931 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : python3-cffi-1.11.5-5.el8.x86_64 27/32 2023-08-26T13:27:27.931 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : python3-pytz-2017.2-9.el8.noarch 28/32 2023-08-26T13:27:27.931 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : python3-markupsafe-0.23-19.el8.x86_64 29/32 2023-08-26T13:27:27.931 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : python3-mako-1.0.6-13.el8.noarch 30/32 2023-08-26T13:27:27.932 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : python3-werkzeug-0.12.2-4.el8.noarch 31/32 2023-08-26T13:27:27.952 INFO:teuthology.orchestra.run.smithi124.stdout: Installing : ceph-2:18.0.0-5706.ge7eb9853.el8.x86_64 32/32 2023-08-26T13:27:28.305 INFO:teuthology.orchestra.run.smithi124.stdout: Running scriptlet: ceph-2:18.0.0-5706.ge7eb9853.el8.x86_64 32/32 2023-08-26T13:27:28.306 INFO:teuthology.orchestra.run.smithi124.stdout: Verifying : ceph-2:18.0.0-5706.ge7eb9853.el8.x86_64 1/32 2023-08-26T13:27:28.306 INFO:teuthology.orchestra.run.smithi124.stdout: Verifying : ceph-mds-2:18.0.0-5706.ge7eb9853.el8.x86_64 2/32 2023-08-26T13:27:28.306 INFO:teuthology.orchestra.run.smithi124.stdout: Verifying : ceph-mgr-2:18.0.0-5706.ge7eb9853.el8.x86_64 3/32 2023-08-26T13:27:28.306 INFO:teuthology.orchestra.run.smithi124.stdout: Verifying : ceph-mon-2:18.0.0-5706.ge7eb9853.el8.x86_64 4/32 2023-08-26T13:27:28.306 INFO:teuthology.orchestra.run.smithi124.stdout: Verifying : ceph-osd-2:18.0.0-5706.ge7eb9853.el8.x86_64 5/32 2023-08-26T13:27:28.307 INFO:teuthology.orchestra.run.smithi124.stdout: Verifying : ceph-mgr-modules-core-2:18.0.0-5706.ge7eb9853.el8. 6/32 2023-08-26T13:27:28.307 INFO:teuthology.orchestra.run.smithi124.stdout: Verifying : python3-bcrypt-3.1.6-2.el8.1.x86_64 7/32 2023-08-26T13:27:28.307 INFO:teuthology.orchestra.run.smithi124.stdout: Verifying : python3-beautifulsoup4-4.6.3-2.el8.1.noarch 8/32 2023-08-26T13:27:28.307 INFO:teuthology.orchestra.run.smithi124.stdout: Verifying : python3-cheroot-8.5.2-1.el8.noarch 9/32 2023-08-26T13:27:28.308 INFO:teuthology.orchestra.run.smithi124.stdout: Verifying : python3-cherrypy-18.4.0-1.el8.noarch 10/32 2023-08-26T13:27:28.308 INFO:teuthology.orchestra.run.smithi124.stdout: Verifying : python3-jaraco-6.2-6.el8.noarch 11/32 2023-08-26T13:27:28.308 INFO:teuthology.orchestra.run.smithi124.stdout: Verifying : python3-jaraco-functools-2.0-4.el8.noarch 12/32 2023-08-26T13:27:28.308 INFO:teuthology.orchestra.run.smithi124.stdout: Verifying : python3-logutils-0.3.5-11.el8.noarch 13/32 2023-08-26T13:27:28.309 INFO:teuthology.orchestra.run.smithi124.stdout: Verifying : python3-more-itertools-7.2.0-3.el8.noarch 14/32 2023-08-26T13:27:28.309 INFO:teuthology.orchestra.run.smithi124.stdout: Verifying : python3-pecan-1.3.2-9.el8.noarch 15/32 2023-08-26T13:27:28.309 INFO:teuthology.orchestra.run.smithi124.stdout: Verifying : python3-portend-2.6-1.el8.noarch 16/32 2023-08-26T13:27:28.309 INFO:teuthology.orchestra.run.smithi124.stdout: Verifying : python3-simplegeneric-0.8.1-17.el8.noarch 17/32 2023-08-26T13:27:28.309 INFO:teuthology.orchestra.run.smithi124.stdout: Verifying : python3-singledispatch-3.4.0.3-18.el8.noarch 18/32 2023-08-26T13:27:28.310 INFO:teuthology.orchestra.run.smithi124.stdout: Verifying : python3-tempora-1.14.1-5.el8.noarch 19/32 2023-08-26T13:27:28.310 INFO:teuthology.orchestra.run.smithi124.stdout: Verifying : python3-trustme-0.6.0-4.el8.noarch 20/32 2023-08-26T13:27:28.310 INFO:teuthology.orchestra.run.smithi124.stdout: Verifying : python3-waitress-1.4.3-1.el8.noarch 21/32 2023-08-26T13:27:28.310 INFO:teuthology.orchestra.run.smithi124.stdout: Verifying : python3-webob-1.8.5-1.el8.1.noarch 22/32 2023-08-26T13:27:28.310 INFO:teuthology.orchestra.run.smithi124.stdout: Verifying : python3-webtest-2.0.33-1.el8.noarch 23/32 2023-08-26T13:27:28.311 INFO:teuthology.orchestra.run.smithi124.stdout: Verifying : python3-zc-lockfile-2.0-2.el8.noarch 24/32 2023-08-26T13:27:28.311 INFO:teuthology.orchestra.run.smithi124.stdout: Verifying : python3-pycparser-2.14-14.el8.noarch 25/32 2023-08-26T13:27:28.311 INFO:teuthology.orchestra.run.smithi124.stdout: Verifying : python3-cryptography-3.2.1-5.el8.x86_64 26/32 2023-08-26T13:27:28.311 INFO:teuthology.orchestra.run.smithi124.stdout: Verifying : python3-cffi-1.11.5-5.el8.x86_64 27/32 2023-08-26T13:27:28.311 INFO:teuthology.orchestra.run.smithi124.stdout: Verifying : python3-pytz-2017.2-9.el8.noarch 28/32 2023-08-26T13:27:28.312 INFO:teuthology.orchestra.run.smithi124.stdout: Verifying : python3-markupsafe-0.23-19.el8.x86_64 29/32 2023-08-26T13:27:28.312 INFO:teuthology.orchestra.run.smithi124.stdout: Verifying : python3-mako-1.0.6-13.el8.noarch 30/32 2023-08-26T13:27:28.312 INFO:teuthology.orchestra.run.smithi124.stdout: Verifying : python3-werkzeug-0.12.2-4.el8.noarch 31/32 2023-08-26T13:27:29.380 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : python3-pyOpenSSL-19.0.0-1.el8.noarch 32/32 2023-08-26T13:27:29.380 INFO:teuthology.orchestra.run.smithi031.stdout:Last metadata expiration check: 0:01:46 ago on Sat 26 Aug 2023 01:25:43 PM UTC. 2023-08-26T13:27:31.429 INFO:teuthology.orchestra.run.smithi124.stdout: Verifying : python3-pyOpenSSL-19.0.0-1.el8.noarch 32/32 2023-08-26T13:27:31.430 INFO:teuthology.orchestra.run.smithi124.stdout:Last metadata expiration check: 0:01:48 ago on Sat 26 Aug 2023 01:25:43 PM UTC. 2023-08-26T13:27:38.042 INFO:teuthology.orchestra.run.smithi031.stdout:Installed products updated. 2023-08-26T13:27:38.249 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-08-26T13:27:38.249 INFO:teuthology.orchestra.run.smithi031.stdout:Installed: 2023-08-26T13:27:38.249 INFO:teuthology.orchestra.run.smithi031.stdout: ceph-2:18.0.0-5706.ge7eb9853.el8.x86_64 2023-08-26T13:27:38.249 INFO:teuthology.orchestra.run.smithi031.stdout: ceph-mds-2:18.0.0-5706.ge7eb9853.el8.x86_64 2023-08-26T13:27:38.250 INFO:teuthology.orchestra.run.smithi031.stdout: ceph-mgr-2:18.0.0-5706.ge7eb9853.el8.x86_64 2023-08-26T13:27:38.250 INFO:teuthology.orchestra.run.smithi031.stdout: ceph-mgr-modules-core-2:18.0.0-5706.ge7eb9853.el8.noarch 2023-08-26T13:27:38.250 INFO:teuthology.orchestra.run.smithi031.stdout: ceph-mon-2:18.0.0-5706.ge7eb9853.el8.x86_64 2023-08-26T13:27:38.250 INFO:teuthology.orchestra.run.smithi031.stdout: ceph-osd-2:18.0.0-5706.ge7eb9853.el8.x86_64 2023-08-26T13:27:38.250 INFO:teuthology.orchestra.run.smithi031.stdout: python3-bcrypt-3.1.6-2.el8.1.x86_64 2023-08-26T13:27:38.251 INFO:teuthology.orchestra.run.smithi031.stdout: python3-beautifulsoup4-4.6.3-2.el8.1.noarch 2023-08-26T13:27:38.251 INFO:teuthology.orchestra.run.smithi031.stdout: python3-cffi-1.11.5-5.el8.x86_64 2023-08-26T13:27:38.251 INFO:teuthology.orchestra.run.smithi031.stdout: python3-cheroot-8.5.2-1.el8.noarch 2023-08-26T13:27:38.251 INFO:teuthology.orchestra.run.smithi031.stdout: python3-cherrypy-18.4.0-1.el8.noarch 2023-08-26T13:27:38.251 INFO:teuthology.orchestra.run.smithi031.stdout: python3-cryptography-3.2.1-5.el8.x86_64 2023-08-26T13:27:38.252 INFO:teuthology.orchestra.run.smithi031.stdout: python3-jaraco-6.2-6.el8.noarch 2023-08-26T13:27:38.252 INFO:teuthology.orchestra.run.smithi031.stdout: python3-jaraco-functools-2.0-4.el8.noarch 2023-08-26T13:27:38.252 INFO:teuthology.orchestra.run.smithi031.stdout: python3-logutils-0.3.5-11.el8.noarch 2023-08-26T13:27:38.252 INFO:teuthology.orchestra.run.smithi031.stdout: python3-mako-1.0.6-13.el8.noarch 2023-08-26T13:27:38.252 INFO:teuthology.orchestra.run.smithi031.stdout: python3-markupsafe-0.23-19.el8.x86_64 2023-08-26T13:27:38.253 INFO:teuthology.orchestra.run.smithi031.stdout: python3-more-itertools-7.2.0-3.el8.noarch 2023-08-26T13:27:38.253 INFO:teuthology.orchestra.run.smithi031.stdout: python3-pecan-1.3.2-9.el8.noarch 2023-08-26T13:27:38.253 INFO:teuthology.orchestra.run.smithi031.stdout: python3-portend-2.6-1.el8.noarch 2023-08-26T13:27:38.253 INFO:teuthology.orchestra.run.smithi031.stdout: python3-pyOpenSSL-19.0.0-1.el8.noarch 2023-08-26T13:27:38.253 INFO:teuthology.orchestra.run.smithi031.stdout: python3-pycparser-2.14-14.el8.noarch 2023-08-26T13:27:38.254 INFO:teuthology.orchestra.run.smithi031.stdout: python3-pytz-2017.2-9.el8.noarch 2023-08-26T13:27:38.254 INFO:teuthology.orchestra.run.smithi031.stdout: python3-simplegeneric-0.8.1-17.el8.noarch 2023-08-26T13:27:38.254 INFO:teuthology.orchestra.run.smithi031.stdout: python3-singledispatch-3.4.0.3-18.el8.noarch 2023-08-26T13:27:38.254 INFO:teuthology.orchestra.run.smithi031.stdout: python3-tempora-1.14.1-5.el8.noarch 2023-08-26T13:27:38.254 INFO:teuthology.orchestra.run.smithi031.stdout: python3-trustme-0.6.0-4.el8.noarch 2023-08-26T13:27:38.255 INFO:teuthology.orchestra.run.smithi031.stdout: python3-waitress-1.4.3-1.el8.noarch 2023-08-26T13:27:38.255 INFO:teuthology.orchestra.run.smithi031.stdout: python3-webob-1.8.5-1.el8.1.noarch 2023-08-26T13:27:38.255 INFO:teuthology.orchestra.run.smithi031.stdout: python3-webtest-2.0.33-1.el8.noarch 2023-08-26T13:27:38.255 INFO:teuthology.orchestra.run.smithi031.stdout: python3-werkzeug-0.12.2-4.el8.noarch 2023-08-26T13:27:38.255 INFO:teuthology.orchestra.run.smithi031.stdout: python3-zc-lockfile-2.0-2.el8.noarch 2023-08-26T13:27:38.256 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-08-26T13:27:38.256 INFO:teuthology.orchestra.run.smithi031.stdout:Complete! 2023-08-26T13:27:38.534 DEBUG:teuthology.orchestra.run.smithi031:> sudo yum -y install ceph-base 2023-08-26T13:27:38.901 INFO:teuthology.orchestra.run.smithi031.stdout:Updating Subscription Management repositories. 2023-08-26T13:27:40.232 INFO:teuthology.orchestra.run.smithi124.stdout:Installed products updated. 2023-08-26T13:27:40.414 INFO:teuthology.orchestra.run.smithi124.stdout: 2023-08-26T13:27:40.415 INFO:teuthology.orchestra.run.smithi124.stdout:Installed: 2023-08-26T13:27:40.415 INFO:teuthology.orchestra.run.smithi124.stdout: ceph-2:18.0.0-5706.ge7eb9853.el8.x86_64 2023-08-26T13:27:40.415 INFO:teuthology.orchestra.run.smithi124.stdout: ceph-mds-2:18.0.0-5706.ge7eb9853.el8.x86_64 2023-08-26T13:27:40.415 INFO:teuthology.orchestra.run.smithi124.stdout: ceph-mgr-2:18.0.0-5706.ge7eb9853.el8.x86_64 2023-08-26T13:27:40.416 INFO:teuthology.orchestra.run.smithi124.stdout: ceph-mgr-modules-core-2:18.0.0-5706.ge7eb9853.el8.noarch 2023-08-26T13:27:40.416 INFO:teuthology.orchestra.run.smithi124.stdout: ceph-mon-2:18.0.0-5706.ge7eb9853.el8.x86_64 2023-08-26T13:27:40.416 INFO:teuthology.orchestra.run.smithi124.stdout: ceph-osd-2:18.0.0-5706.ge7eb9853.el8.x86_64 2023-08-26T13:27:40.416 INFO:teuthology.orchestra.run.smithi124.stdout: python3-bcrypt-3.1.6-2.el8.1.x86_64 2023-08-26T13:27:40.416 INFO:teuthology.orchestra.run.smithi124.stdout: python3-beautifulsoup4-4.6.3-2.el8.1.noarch 2023-08-26T13:27:40.417 INFO:teuthology.orchestra.run.smithi124.stdout: python3-cffi-1.11.5-5.el8.x86_64 2023-08-26T13:27:40.417 INFO:teuthology.orchestra.run.smithi124.stdout: python3-cheroot-8.5.2-1.el8.noarch 2023-08-26T13:27:40.417 INFO:teuthology.orchestra.run.smithi124.stdout: python3-cherrypy-18.4.0-1.el8.noarch 2023-08-26T13:27:40.417 INFO:teuthology.orchestra.run.smithi124.stdout: python3-cryptography-3.2.1-5.el8.x86_64 2023-08-26T13:27:40.417 INFO:teuthology.orchestra.run.smithi124.stdout: python3-jaraco-6.2-6.el8.noarch 2023-08-26T13:27:40.418 INFO:teuthology.orchestra.run.smithi124.stdout: python3-jaraco-functools-2.0-4.el8.noarch 2023-08-26T13:27:40.418 INFO:teuthology.orchestra.run.smithi124.stdout: python3-logutils-0.3.5-11.el8.noarch 2023-08-26T13:27:40.418 INFO:teuthology.orchestra.run.smithi124.stdout: python3-mako-1.0.6-13.el8.noarch 2023-08-26T13:27:40.418 INFO:teuthology.orchestra.run.smithi124.stdout: python3-markupsafe-0.23-19.el8.x86_64 2023-08-26T13:27:40.418 INFO:teuthology.orchestra.run.smithi124.stdout: python3-more-itertools-7.2.0-3.el8.noarch 2023-08-26T13:27:40.418 INFO:teuthology.orchestra.run.smithi124.stdout: python3-pecan-1.3.2-9.el8.noarch 2023-08-26T13:27:40.419 INFO:teuthology.orchestra.run.smithi124.stdout: python3-portend-2.6-1.el8.noarch 2023-08-26T13:27:40.419 INFO:teuthology.orchestra.run.smithi124.stdout: python3-pyOpenSSL-19.0.0-1.el8.noarch 2023-08-26T13:27:40.419 INFO:teuthology.orchestra.run.smithi124.stdout: python3-pycparser-2.14-14.el8.noarch 2023-08-26T13:27:40.419 INFO:teuthology.orchestra.run.smithi124.stdout: python3-pytz-2017.2-9.el8.noarch 2023-08-26T13:27:40.419 INFO:teuthology.orchestra.run.smithi124.stdout: python3-simplegeneric-0.8.1-17.el8.noarch 2023-08-26T13:27:40.420 INFO:teuthology.orchestra.run.smithi124.stdout: python3-singledispatch-3.4.0.3-18.el8.noarch 2023-08-26T13:27:40.420 INFO:teuthology.orchestra.run.smithi124.stdout: python3-tempora-1.14.1-5.el8.noarch 2023-08-26T13:27:40.420 INFO:teuthology.orchestra.run.smithi124.stdout: python3-trustme-0.6.0-4.el8.noarch 2023-08-26T13:27:40.420 INFO:teuthology.orchestra.run.smithi124.stdout: python3-waitress-1.4.3-1.el8.noarch 2023-08-26T13:27:40.420 INFO:teuthology.orchestra.run.smithi124.stdout: python3-webob-1.8.5-1.el8.1.noarch 2023-08-26T13:27:40.420 INFO:teuthology.orchestra.run.smithi124.stdout: python3-webtest-2.0.33-1.el8.noarch 2023-08-26T13:27:40.421 INFO:teuthology.orchestra.run.smithi124.stdout: python3-werkzeug-0.12.2-4.el8.noarch 2023-08-26T13:27:40.421 INFO:teuthology.orchestra.run.smithi124.stdout: python3-zc-lockfile-2.0-2.el8.noarch 2023-08-26T13:27:40.421 INFO:teuthology.orchestra.run.smithi124.stdout: 2023-08-26T13:27:40.421 INFO:teuthology.orchestra.run.smithi124.stdout:Complete! 2023-08-26T13:27:40.427 INFO:teuthology.orchestra.run.smithi031.stdout:Last metadata expiration check: 0:01:57 ago on Sat 26 Aug 2023 01:25:43 PM UTC. 2023-08-26T13:27:40.572 DEBUG:teuthology.orchestra.run.smithi124:> sudo yum -y install ceph-base 2023-08-26T13:27:40.918 INFO:teuthology.orchestra.run.smithi124.stdout:Updating Subscription Management repositories. 2023-08-26T13:27:41.836 INFO:teuthology.orchestra.run.smithi031.stdout:Package ceph-base-2:18.0.0-5706.ge7eb9853.el8.x86_64 is already installed. 2023-08-26T13:27:41.953 INFO:teuthology.orchestra.run.smithi031.stdout:Dependencies resolved. 2023-08-26T13:27:41.954 INFO:teuthology.orchestra.run.smithi031.stdout:Nothing to do. 2023-08-26T13:27:41.954 INFO:teuthology.orchestra.run.smithi031.stdout:Complete! 2023-08-26T13:27:42.067 DEBUG:teuthology.orchestra.run.smithi031:> sudo yum -y install cephadm 2023-08-26T13:27:42.431 INFO:teuthology.orchestra.run.smithi031.stdout:Updating Subscription Management repositories. 2023-08-26T13:27:42.574 INFO:teuthology.orchestra.run.smithi124.stdout:Last metadata expiration check: 0:01:59 ago on Sat 26 Aug 2023 01:25:43 PM UTC. 2023-08-26T13:27:43.851 INFO:teuthology.orchestra.run.smithi124.stdout:Package ceph-base-2:18.0.0-5706.ge7eb9853.el8.x86_64 is already installed. 2023-08-26T13:27:43.961 INFO:teuthology.orchestra.run.smithi124.stdout:Dependencies resolved. 2023-08-26T13:27:43.962 INFO:teuthology.orchestra.run.smithi124.stdout:Nothing to do. 2023-08-26T13:27:43.963 INFO:teuthology.orchestra.run.smithi124.stdout:Complete! 2023-08-26T13:27:44.075 DEBUG:teuthology.orchestra.run.smithi124:> sudo yum -y install cephadm 2023-08-26T13:27:44.356 INFO:teuthology.orchestra.run.smithi031.stdout:Last metadata expiration check: 0:02:01 ago on Sat 26 Aug 2023 01:25:43 PM UTC. 2023-08-26T13:27:44.416 INFO:teuthology.orchestra.run.smithi124.stdout:Updating Subscription Management repositories. 2023-08-26T13:27:45.909 INFO:teuthology.orchestra.run.smithi031.stdout:Dependencies resolved. 2023-08-26T13:27:45.910 INFO:teuthology.orchestra.run.smithi031.stdout:================================================================================ 2023-08-26T13:27:45.911 INFO:teuthology.orchestra.run.smithi031.stdout: Package Arch Version Repository Size 2023-08-26T13:27:45.911 INFO:teuthology.orchestra.run.smithi031.stdout:================================================================================ 2023-08-26T13:27:45.911 INFO:teuthology.orchestra.run.smithi031.stdout:Installing: 2023-08-26T13:27:45.911 INFO:teuthology.orchestra.run.smithi031.stdout: cephadm noarch 2:18.0.0-5706.ge7eb9853.el8 ceph-noarch 171 k 2023-08-26T13:27:45.911 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-08-26T13:27:45.912 INFO:teuthology.orchestra.run.smithi031.stdout:Transaction Summary 2023-08-26T13:27:45.912 INFO:teuthology.orchestra.run.smithi031.stdout:================================================================================ 2023-08-26T13:27:45.912 INFO:teuthology.orchestra.run.smithi031.stdout:Install 1 Package 2023-08-26T13:27:45.912 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-08-26T13:27:45.912 INFO:teuthology.orchestra.run.smithi031.stdout:Total download size: 171 k 2023-08-26T13:27:45.913 INFO:teuthology.orchestra.run.smithi031.stdout:Installed size: 694 k 2023-08-26T13:27:45.913 INFO:teuthology.orchestra.run.smithi031.stdout:Downloading Packages: 2023-08-26T13:27:46.079 INFO:teuthology.orchestra.run.smithi124.stdout:Last metadata expiration check: 0:02:03 ago on Sat 26 Aug 2023 01:25:43 PM UTC. 2023-08-26T13:27:46.425 INFO:teuthology.orchestra.run.smithi031.stdout:cephadm-18.0.0-5706.ge7eb9853.el8.noarch.rpm 333 kB/s | 171 kB 00:00 2023-08-26T13:27:46.425 INFO:teuthology.orchestra.run.smithi031.stdout:-------------------------------------------------------------------------------- 2023-08-26T13:27:46.426 INFO:teuthology.orchestra.run.smithi031.stdout:Total 332 kB/s | 171 kB 00:00 2023-08-26T13:27:46.426 INFO:teuthology.orchestra.run.smithi031.stdout:Running transaction check 2023-08-26T13:27:46.435 INFO:teuthology.orchestra.run.smithi031.stdout:Transaction check succeeded. 2023-08-26T13:27:46.436 INFO:teuthology.orchestra.run.smithi031.stdout:Running transaction test 2023-08-26T13:27:46.438 INFO:teuthology.orchestra.run.smithi031.stdout:Transaction test succeeded. 2023-08-26T13:27:46.442 INFO:teuthology.orchestra.run.smithi031.stdout:Running transaction 2023-08-26T13:27:46.486 INFO:teuthology.orchestra.run.smithi031.stdout: Preparing : 1/1 2023-08-26T13:27:47.448 INFO:teuthology.orchestra.run.smithi124.stdout:Dependencies resolved. 2023-08-26T13:27:47.449 INFO:teuthology.orchestra.run.smithi124.stdout:================================================================================ 2023-08-26T13:27:47.449 INFO:teuthology.orchestra.run.smithi124.stdout: Package Arch Version Repository Size 2023-08-26T13:27:47.449 INFO:teuthology.orchestra.run.smithi124.stdout:================================================================================ 2023-08-26T13:27:47.450 INFO:teuthology.orchestra.run.smithi124.stdout:Installing: 2023-08-26T13:27:47.450 INFO:teuthology.orchestra.run.smithi124.stdout: cephadm noarch 2:18.0.0-5706.ge7eb9853.el8 ceph-noarch 171 k 2023-08-26T13:27:47.450 INFO:teuthology.orchestra.run.smithi124.stdout: 2023-08-26T13:27:47.450 INFO:teuthology.orchestra.run.smithi124.stdout:Transaction Summary 2023-08-26T13:27:47.450 INFO:teuthology.orchestra.run.smithi124.stdout:================================================================================ 2023-08-26T13:27:47.450 INFO:teuthology.orchestra.run.smithi124.stdout:Install 1 Package 2023-08-26T13:27:47.451 INFO:teuthology.orchestra.run.smithi124.stdout: 2023-08-26T13:27:47.451 INFO:teuthology.orchestra.run.smithi124.stdout:Total download size: 171 k 2023-08-26T13:27:47.451 INFO:teuthology.orchestra.run.smithi124.stdout:Installed size: 694 k 2023-08-26T13:27:47.451 INFO:teuthology.orchestra.run.smithi124.stdout:Downloading Packages: 2023-08-26T13:27:47.924 INFO:teuthology.orchestra.run.smithi124.stdout:cephadm-18.0.0-5706.ge7eb9853.el8.noarch.rpm 361 kB/s | 171 kB 00:00 2023-08-26T13:27:47.925 INFO:teuthology.orchestra.run.smithi124.stdout:-------------------------------------------------------------------------------- 2023-08-26T13:27:47.925 INFO:teuthology.orchestra.run.smithi124.stdout:Total 360 kB/s | 171 kB 00:00 2023-08-26T13:27:47.925 INFO:teuthology.orchestra.run.smithi124.stdout:Running transaction check 2023-08-26T13:27:47.932 INFO:teuthology.orchestra.run.smithi124.stdout:Transaction check succeeded. 2023-08-26T13:27:47.933 INFO:teuthology.orchestra.run.smithi124.stdout:Running transaction test 2023-08-26T13:27:47.935 INFO:teuthology.orchestra.run.smithi124.stdout:Transaction test succeeded. 2023-08-26T13:27:47.938 INFO:teuthology.orchestra.run.smithi124.stdout:Running transaction 2023-08-26T13:27:47.978 INFO:teuthology.orchestra.run.smithi124.stdout: Preparing : 1/1 2023-08-26T13:27:48.289 INFO:teuthology.orchestra.run.smithi031.stdout: Running scriptlet: cephadm-2:18.0.0-5706.ge7eb9853.el8.noarch 1/1 2023-08-26T13:27:48.437 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : cephadm-2:18.0.0-5706.ge7eb9853.el8.noarch 1/1 2023-08-26T13:27:48.612 INFO:teuthology.orchestra.run.smithi031.stdout: Running scriptlet: cephadm-2:18.0.0-5706.ge7eb9853.el8.noarch 1/1 2023-08-26T13:27:49.654 INFO:teuthology.orchestra.run.smithi124.stdout: Running scriptlet: cephadm-2:18.0.0-5706.ge7eb9853.el8.noarch 1/1 2023-08-26T13:27:49.710 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : cephadm-2:18.0.0-5706.ge7eb9853.el8.noarch 1/1 2023-08-26T13:27:49.710 INFO:teuthology.orchestra.run.smithi031.stdout:Last metadata expiration check: 0:02:06 ago on Sat 26 Aug 2023 01:25:43 PM UTC. 2023-08-26T13:27:49.803 INFO:teuthology.orchestra.run.smithi124.stdout: Installing : cephadm-2:18.0.0-5706.ge7eb9853.el8.noarch 1/1 2023-08-26T13:27:49.968 INFO:teuthology.orchestra.run.smithi124.stdout: Running scriptlet: cephadm-2:18.0.0-5706.ge7eb9853.el8.noarch 1/1 2023-08-26T13:27:50.993 INFO:teuthology.orchestra.run.smithi124.stdout: Verifying : cephadm-2:18.0.0-5706.ge7eb9853.el8.noarch 1/1 2023-08-26T13:27:50.994 INFO:teuthology.orchestra.run.smithi124.stdout:Last metadata expiration check: 0:02:07 ago on Sat 26 Aug 2023 01:25:43 PM UTC. 2023-08-26T13:27:59.960 INFO:teuthology.orchestra.run.smithi031.stdout:Installed products updated. 2023-08-26T13:28:00.159 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-08-26T13:28:00.159 INFO:teuthology.orchestra.run.smithi031.stdout:Installed: 2023-08-26T13:28:00.159 INFO:teuthology.orchestra.run.smithi031.stdout: cephadm-2:18.0.0-5706.ge7eb9853.el8.noarch 2023-08-26T13:28:00.159 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-08-26T13:28:00.159 INFO:teuthology.orchestra.run.smithi031.stdout:Complete! 2023-08-26T13:28:00.449 DEBUG:teuthology.orchestra.run.smithi031:> sudo yum -y install ceph-immutable-object-cache 2023-08-26T13:28:00.827 INFO:teuthology.orchestra.run.smithi031.stdout:Updating Subscription Management repositories. 2023-08-26T13:28:01.066 INFO:teuthology.orchestra.run.smithi124.stdout:Installed products updated. 2023-08-26T13:28:01.247 INFO:teuthology.orchestra.run.smithi124.stdout: 2023-08-26T13:28:01.248 INFO:teuthology.orchestra.run.smithi124.stdout:Installed: 2023-08-26T13:28:01.248 INFO:teuthology.orchestra.run.smithi124.stdout: cephadm-2:18.0.0-5706.ge7eb9853.el8.noarch 2023-08-26T13:28:01.248 INFO:teuthology.orchestra.run.smithi124.stdout: 2023-08-26T13:28:01.248 INFO:teuthology.orchestra.run.smithi124.stdout:Complete! 2023-08-26T13:28:01.385 DEBUG:teuthology.orchestra.run.smithi124:> sudo yum -y install ceph-immutable-object-cache 2023-08-26T13:28:01.744 INFO:teuthology.orchestra.run.smithi124.stdout:Updating Subscription Management repositories. 2023-08-26T13:28:02.499 INFO:teuthology.orchestra.run.smithi031.stdout:Last metadata expiration check: 0:02:19 ago on Sat 26 Aug 2023 01:25:43 PM UTC. 2023-08-26T13:28:03.338 INFO:teuthology.orchestra.run.smithi124.stdout:Last metadata expiration check: 0:02:20 ago on Sat 26 Aug 2023 01:25:43 PM UTC. 2023-08-26T13:28:04.034 INFO:teuthology.orchestra.run.smithi031.stdout:Dependencies resolved. 2023-08-26T13:28:04.035 INFO:teuthology.orchestra.run.smithi031.stdout:================================================================================ 2023-08-26T13:28:04.035 INFO:teuthology.orchestra.run.smithi031.stdout: Package Arch Version Repo Size 2023-08-26T13:28:04.036 INFO:teuthology.orchestra.run.smithi031.stdout:================================================================================ 2023-08-26T13:28:04.036 INFO:teuthology.orchestra.run.smithi031.stdout:Installing: 2023-08-26T13:28:04.036 INFO:teuthology.orchestra.run.smithi031.stdout: ceph-immutable-object-cache x86_64 2:18.0.0-5706.ge7eb9853.el8 ceph 167 k 2023-08-26T13:28:04.036 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-08-26T13:28:04.037 INFO:teuthology.orchestra.run.smithi031.stdout:Transaction Summary 2023-08-26T13:28:04.037 INFO:teuthology.orchestra.run.smithi031.stdout:================================================================================ 2023-08-26T13:28:04.037 INFO:teuthology.orchestra.run.smithi031.stdout:Install 1 Package 2023-08-26T13:28:04.037 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-08-26T13:28:04.037 INFO:teuthology.orchestra.run.smithi031.stdout:Total download size: 167 k 2023-08-26T13:28:04.038 INFO:teuthology.orchestra.run.smithi031.stdout:Installed size: 512 k 2023-08-26T13:28:04.038 INFO:teuthology.orchestra.run.smithi031.stdout:Downloading Packages: 2023-08-26T13:28:04.414 INFO:teuthology.orchestra.run.smithi031.stdout:ceph-immutable-object-cache-18.0.0-5706.ge7eb98 441 kB/s | 167 kB 00:00 2023-08-26T13:28:04.415 INFO:teuthology.orchestra.run.smithi031.stdout:-------------------------------------------------------------------------------- 2023-08-26T13:28:04.415 INFO:teuthology.orchestra.run.smithi031.stdout:Total 440 kB/s | 167 kB 00:00 2023-08-26T13:28:04.415 INFO:teuthology.orchestra.run.smithi031.stdout:Running transaction check 2023-08-26T13:28:04.427 INFO:teuthology.orchestra.run.smithi031.stdout:Transaction check succeeded. 2023-08-26T13:28:04.428 INFO:teuthology.orchestra.run.smithi031.stdout:Running transaction test 2023-08-26T13:28:04.484 INFO:teuthology.orchestra.run.smithi031.stdout:Transaction test succeeded. 2023-08-26T13:28:04.488 INFO:teuthology.orchestra.run.smithi031.stdout:Running transaction 2023-08-26T13:28:04.614 INFO:teuthology.orchestra.run.smithi031.stdout: Preparing : 1/1 2023-08-26T13:28:04.703 INFO:teuthology.orchestra.run.smithi124.stdout:Dependencies resolved. 2023-08-26T13:28:04.704 INFO:teuthology.orchestra.run.smithi124.stdout:================================================================================ 2023-08-26T13:28:04.704 INFO:teuthology.orchestra.run.smithi124.stdout: Package Arch Version Repo Size 2023-08-26T13:28:04.705 INFO:teuthology.orchestra.run.smithi124.stdout:================================================================================ 2023-08-26T13:28:04.705 INFO:teuthology.orchestra.run.smithi124.stdout:Installing: 2023-08-26T13:28:04.705 INFO:teuthology.orchestra.run.smithi124.stdout: ceph-immutable-object-cache x86_64 2:18.0.0-5706.ge7eb9853.el8 ceph 167 k 2023-08-26T13:28:04.705 INFO:teuthology.orchestra.run.smithi124.stdout: 2023-08-26T13:28:04.705 INFO:teuthology.orchestra.run.smithi124.stdout:Transaction Summary 2023-08-26T13:28:04.705 INFO:teuthology.orchestra.run.smithi124.stdout:================================================================================ 2023-08-26T13:28:04.706 INFO:teuthology.orchestra.run.smithi124.stdout:Install 1 Package 2023-08-26T13:28:04.706 INFO:teuthology.orchestra.run.smithi124.stdout: 2023-08-26T13:28:04.706 INFO:teuthology.orchestra.run.smithi124.stdout:Total download size: 167 k 2023-08-26T13:28:04.706 INFO:teuthology.orchestra.run.smithi124.stdout:Installed size: 512 k 2023-08-26T13:28:04.706 INFO:teuthology.orchestra.run.smithi124.stdout:Downloading Packages: 2023-08-26T13:28:04.890 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : ceph-immutable-object-cache-2:18.0.0-5706.ge7eb9853. 1/1 2023-08-26T13:28:04.893 INFO:teuthology.orchestra.run.smithi124.stdout:ceph-immutable-object-cache-18.0.0-5706.ge7eb98 889 kB/s | 167 kB 00:00 2023-08-26T13:28:04.893 INFO:teuthology.orchestra.run.smithi124.stdout:-------------------------------------------------------------------------------- 2023-08-26T13:28:04.894 INFO:teuthology.orchestra.run.smithi124.stdout:Total 883 kB/s | 167 kB 00:00 2023-08-26T13:28:04.894 INFO:teuthology.orchestra.run.smithi124.stdout:Running transaction check 2023-08-26T13:28:04.905 INFO:teuthology.orchestra.run.smithi124.stdout:Transaction check succeeded. 2023-08-26T13:28:04.906 INFO:teuthology.orchestra.run.smithi124.stdout:Running transaction test 2023-08-26T13:28:04.957 INFO:teuthology.orchestra.run.smithi124.stdout:Transaction test succeeded. 2023-08-26T13:28:04.960 INFO:teuthology.orchestra.run.smithi124.stdout:Running transaction 2023-08-26T13:28:05.075 INFO:teuthology.orchestra.run.smithi124.stdout: Preparing : 1/1 2023-08-26T13:28:05.216 INFO:teuthology.orchestra.run.smithi124.stdout: Installing : ceph-immutable-object-cache-2:18.0.0-5706.ge7eb9853. 1/1 2023-08-26T13:28:05.261 INFO:teuthology.orchestra.run.smithi031.stdout: Running scriptlet: ceph-immutable-object-cache-2:18.0.0-5706.ge7eb9853. 1/1 2023-08-26T13:28:05.568 INFO:teuthology.orchestra.run.smithi124.stdout: Running scriptlet: ceph-immutable-object-cache-2:18.0.0-5706.ge7eb9853. 1/1 2023-08-26T13:28:06.328 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : ceph-immutable-object-cache-2:18.0.0-5706.ge7eb9853. 1/1 2023-08-26T13:28:06.328 INFO:teuthology.orchestra.run.smithi031.stdout:Last metadata expiration check: 0:02:23 ago on Sat 26 Aug 2023 01:25:43 PM UTC. 2023-08-26T13:28:06.586 INFO:teuthology.orchestra.run.smithi124.stdout: Verifying : ceph-immutable-object-cache-2:18.0.0-5706.ge7eb9853. 1/1 2023-08-26T13:28:06.586 INFO:teuthology.orchestra.run.smithi124.stdout:Last metadata expiration check: 0:02:23 ago on Sat 26 Aug 2023 01:25:43 PM UTC. 2023-08-26T13:28:15.985 INFO:teuthology.orchestra.run.smithi031.stdout:Installed products updated. 2023-08-26T13:28:16.169 INFO:teuthology.orchestra.run.smithi124.stdout:Installed products updated. 2023-08-26T13:28:16.186 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-08-26T13:28:16.186 INFO:teuthology.orchestra.run.smithi031.stdout:Installed: 2023-08-26T13:28:16.186 INFO:teuthology.orchestra.run.smithi031.stdout: ceph-immutable-object-cache-2:18.0.0-5706.ge7eb9853.el8.x86_64 2023-08-26T13:28:16.186 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-08-26T13:28:16.187 INFO:teuthology.orchestra.run.smithi031.stdout:Complete! 2023-08-26T13:28:16.312 DEBUG:teuthology.orchestra.run.smithi031:> sudo yum -y install ceph-mgr 2023-08-26T13:28:16.356 INFO:teuthology.orchestra.run.smithi124.stdout: 2023-08-26T13:28:16.356 INFO:teuthology.orchestra.run.smithi124.stdout:Installed: 2023-08-26T13:28:16.356 INFO:teuthology.orchestra.run.smithi124.stdout: ceph-immutable-object-cache-2:18.0.0-5706.ge7eb9853.el8.x86_64 2023-08-26T13:28:16.356 INFO:teuthology.orchestra.run.smithi124.stdout: 2023-08-26T13:28:16.356 INFO:teuthology.orchestra.run.smithi124.stdout:Complete! 2023-08-26T13:28:16.472 DEBUG:teuthology.orchestra.run.smithi124:> sudo yum -y install ceph-mgr 2023-08-26T13:28:16.722 INFO:teuthology.orchestra.run.smithi031.stdout:Updating Subscription Management repositories. 2023-08-26T13:28:16.815 INFO:teuthology.orchestra.run.smithi124.stdout:Updating Subscription Management repositories. 2023-08-26T13:28:18.062 INFO:teuthology.orchestra.run.smithi031.stdout:Last metadata expiration check: 0:02:35 ago on Sat 26 Aug 2023 01:25:43 PM UTC. 2023-08-26T13:28:18.312 INFO:teuthology.orchestra.run.smithi124.stdout:Last metadata expiration check: 0:02:35 ago on Sat 26 Aug 2023 01:25:43 PM UTC. 2023-08-26T13:28:19.505 INFO:teuthology.orchestra.run.smithi031.stdout:Package ceph-mgr-2:18.0.0-5706.ge7eb9853.el8.x86_64 is already installed. 2023-08-26T13:28:19.600 INFO:teuthology.orchestra.run.smithi124.stdout:Package ceph-mgr-2:18.0.0-5706.ge7eb9853.el8.x86_64 is already installed. 2023-08-26T13:28:19.624 INFO:teuthology.orchestra.run.smithi031.stdout:Dependencies resolved. 2023-08-26T13:28:19.625 INFO:teuthology.orchestra.run.smithi031.stdout:Nothing to do. 2023-08-26T13:28:19.626 INFO:teuthology.orchestra.run.smithi031.stdout:Complete! 2023-08-26T13:28:19.708 INFO:teuthology.orchestra.run.smithi124.stdout:Dependencies resolved. 2023-08-26T13:28:19.709 INFO:teuthology.orchestra.run.smithi124.stdout:Nothing to do. 2023-08-26T13:28:19.709 INFO:teuthology.orchestra.run.smithi124.stdout:Complete! 2023-08-26T13:28:19.735 DEBUG:teuthology.orchestra.run.smithi031:> sudo yum -y install ceph-mgr-dashboard 2023-08-26T13:28:19.811 DEBUG:teuthology.orchestra.run.smithi124:> sudo yum -y install ceph-mgr-dashboard 2023-08-26T13:28:20.097 INFO:teuthology.orchestra.run.smithi031.stdout:Updating Subscription Management repositories. 2023-08-26T13:28:20.153 INFO:teuthology.orchestra.run.smithi124.stdout:Updating Subscription Management repositories. 2023-08-26T13:28:21.778 INFO:teuthology.orchestra.run.smithi031.stdout:Last metadata expiration check: 0:02:38 ago on Sat 26 Aug 2023 01:25:43 PM UTC. 2023-08-26T13:28:21.809 INFO:teuthology.orchestra.run.smithi124.stdout:Last metadata expiration check: 0:02:38 ago on Sat 26 Aug 2023 01:25:43 PM UTC. 2023-08-26T13:28:23.265 INFO:teuthology.orchestra.run.smithi124.stdout:Dependencies resolved. 2023-08-26T13:28:23.267 INFO:teuthology.orchestra.run.smithi124.stdout:=================================================================================================== 2023-08-26T13:28:23.267 INFO:teuthology.orchestra.run.smithi124.stdout: Package Arch Version Repository Size 2023-08-26T13:28:23.267 INFO:teuthology.orchestra.run.smithi124.stdout:=================================================================================================== 2023-08-26T13:28:23.267 INFO:teuthology.orchestra.run.smithi124.stdout:Installing: 2023-08-26T13:28:23.267 INFO:teuthology.orchestra.run.smithi124.stdout: ceph-mgr-dashboard noarch 2:18.0.0-5706.ge7eb9853.el8 ceph-noarch 1.7 M 2023-08-26T13:28:23.268 INFO:teuthology.orchestra.run.smithi124.stdout:Installing dependencies: 2023-08-26T13:28:23.268 INFO:teuthology.orchestra.run.smithi124.stdout: ceph-grafana-dashboards noarch 2:18.0.0-5706.ge7eb9853.el8 ceph-noarch 23 k 2023-08-26T13:28:23.268 INFO:teuthology.orchestra.run.smithi124.stdout: ceph-prometheus-alerts noarch 2:18.0.0-5706.ge7eb9853.el8 ceph-noarch 15 k 2023-08-26T13:28:23.268 INFO:teuthology.orchestra.run.smithi124.stdout: python3-jwt noarch 1.6.1-2.el8 rhel-8-for-x86_64-baseos-rpms 43 k 2023-08-26T13:28:23.268 INFO:teuthology.orchestra.run.smithi124.stdout: python3-repoze-lru noarch 0.7-6.el8 epel 34 k 2023-08-26T13:28:23.269 INFO:teuthology.orchestra.run.smithi124.stdout: python3-routes noarch 2.4.1-12.el8 epel 196 k 2023-08-26T13:28:23.269 INFO:teuthology.orchestra.run.smithi124.stdout: 2023-08-26T13:28:23.269 INFO:teuthology.orchestra.run.smithi124.stdout:Transaction Summary 2023-08-26T13:28:23.269 INFO:teuthology.orchestra.run.smithi124.stdout:=================================================================================================== 2023-08-26T13:28:23.269 INFO:teuthology.orchestra.run.smithi124.stdout:Install 6 Packages 2023-08-26T13:28:23.270 INFO:teuthology.orchestra.run.smithi124.stdout: 2023-08-26T13:28:23.270 INFO:teuthology.orchestra.run.smithi124.stdout:Total download size: 2.1 M 2023-08-26T13:28:23.270 INFO:teuthology.orchestra.run.smithi124.stdout:Installed size: 8.5 M 2023-08-26T13:28:23.271 INFO:teuthology.orchestra.run.smithi124.stdout:Downloading Packages: 2023-08-26T13:28:23.351 INFO:teuthology.orchestra.run.smithi031.stdout:Dependencies resolved. 2023-08-26T13:28:23.352 INFO:teuthology.orchestra.run.smithi031.stdout:=================================================================================================== 2023-08-26T13:28:23.352 INFO:teuthology.orchestra.run.smithi031.stdout: Package Arch Version Repository Size 2023-08-26T13:28:23.352 INFO:teuthology.orchestra.run.smithi031.stdout:=================================================================================================== 2023-08-26T13:28:23.353 INFO:teuthology.orchestra.run.smithi031.stdout:Installing: 2023-08-26T13:28:23.353 INFO:teuthology.orchestra.run.smithi031.stdout: ceph-mgr-dashboard noarch 2:18.0.0-5706.ge7eb9853.el8 ceph-noarch 1.7 M 2023-08-26T13:28:23.353 INFO:teuthology.orchestra.run.smithi031.stdout:Installing dependencies: 2023-08-26T13:28:23.353 INFO:teuthology.orchestra.run.smithi031.stdout: ceph-grafana-dashboards noarch 2:18.0.0-5706.ge7eb9853.el8 ceph-noarch 23 k 2023-08-26T13:28:23.353 INFO:teuthology.orchestra.run.smithi031.stdout: ceph-prometheus-alerts noarch 2:18.0.0-5706.ge7eb9853.el8 ceph-noarch 15 k 2023-08-26T13:28:23.354 INFO:teuthology.orchestra.run.smithi031.stdout: python3-jwt noarch 1.6.1-2.el8 rhel-8-for-x86_64-baseos-rpms 43 k 2023-08-26T13:28:23.354 INFO:teuthology.orchestra.run.smithi031.stdout: python3-repoze-lru noarch 0.7-6.el8 epel 34 k 2023-08-26T13:28:23.354 INFO:teuthology.orchestra.run.smithi031.stdout: python3-routes noarch 2.4.1-12.el8 epel 196 k 2023-08-26T13:28:23.354 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-08-26T13:28:23.354 INFO:teuthology.orchestra.run.smithi031.stdout:Transaction Summary 2023-08-26T13:28:23.354 INFO:teuthology.orchestra.run.smithi031.stdout:=================================================================================================== 2023-08-26T13:28:23.355 INFO:teuthology.orchestra.run.smithi031.stdout:Install 6 Packages 2023-08-26T13:28:23.355 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-08-26T13:28:23.356 INFO:teuthology.orchestra.run.smithi031.stdout:Total download size: 2.1 M 2023-08-26T13:28:23.356 INFO:teuthology.orchestra.run.smithi031.stdout:Installed size: 8.5 M 2023-08-26T13:28:23.356 INFO:teuthology.orchestra.run.smithi031.stdout:Downloading Packages: 2023-08-26T13:28:23.456 INFO:teuthology.orchestra.run.smithi124.stdout:(1/6): ceph-grafana-dashboards-18.0.0-5706.ge7e 150 kB/s | 23 kB 00:00 2023-08-26T13:28:23.506 INFO:teuthology.orchestra.run.smithi124.stdout:(2/6): ceph-prometheus-alerts-18.0.0-5706.ge7eb 71 kB/s | 15 kB 00:00 2023-08-26T13:28:23.623 INFO:teuthology.orchestra.run.smithi124.stdout:(3/6): ceph-mgr-dashboard-18.0.0-5706.ge7eb9853 5.5 MB/s | 1.7 MB 00:00 2023-08-26T13:28:23.648 INFO:teuthology.orchestra.run.smithi124.stdout:(4/6): python3-repoze-lru-0.7-6.el8.noarch.rpm 175 kB/s | 34 kB 00:00 2023-08-26T13:28:23.765 INFO:teuthology.orchestra.run.smithi124.stdout:(5/6): python3-routes-2.4.1-12.el8.noarch.rpm 759 kB/s | 196 kB 00:00 2023-08-26T13:28:23.837 INFO:teuthology.orchestra.run.smithi124.stdout:(6/6): python3-jwt-1.6.1-2.el8.noarch.rpm 200 kB/s | 43 kB 00:00 2023-08-26T13:28:23.838 INFO:teuthology.orchestra.run.smithi124.stdout:-------------------------------------------------------------------------------- 2023-08-26T13:28:23.839 INFO:teuthology.orchestra.run.smithi124.stdout:Total 3.6 MB/s | 2.1 MB 00:00 2023-08-26T13:28:23.850 INFO:teuthology.orchestra.run.smithi124.stdout:Running transaction check 2023-08-26T13:28:23.859 INFO:teuthology.orchestra.run.smithi124.stdout:Transaction check succeeded. 2023-08-26T13:28:23.860 INFO:teuthology.orchestra.run.smithi124.stdout:Running transaction test 2023-08-26T13:28:23.873 INFO:teuthology.orchestra.run.smithi031.stdout:(1/6): ceph-prometheus-alerts-18.0.0-5706.ge7eb 103 kB/s | 15 kB 00:00 2023-08-26T13:28:23.916 INFO:teuthology.orchestra.run.smithi124.stdout:Transaction test succeeded. 2023-08-26T13:28:23.920 INFO:teuthology.orchestra.run.smithi124.stdout:Running transaction 2023-08-26T13:28:23.965 INFO:teuthology.orchestra.run.smithi031.stdout:(2/6): ceph-grafana-dashboards-18.0.0-5706.ge7e 99 kB/s | 23 kB 00:00 2023-08-26T13:28:24.249 INFO:teuthology.orchestra.run.smithi031.stdout:(3/6): ceph-mgr-dashboard-18.0.0-5706.ge7eb9853 3.4 MB/s | 1.7 MB 00:00 2023-08-26T13:28:24.367 INFO:teuthology.orchestra.run.smithi124.stdout: Preparing : 1/1 2023-08-26T13:28:24.424 INFO:teuthology.orchestra.run.smithi031.stdout:(4/6): python3-repoze-lru-0.7-6.el8.noarch.rpm 61 kB/s | 34 kB 00:00 2023-08-26T13:28:24.449 INFO:teuthology.orchestra.run.smithi031.stdout:(5/6): python3-jwt-1.6.1-2.el8.noarch.rpm 214 kB/s | 43 kB 00:00 2023-08-26T13:28:24.564 INFO:teuthology.orchestra.run.smithi124.stdout: Installing : python3-jwt-1.6.1-2.el8.noarch 1/6 2023-08-26T13:28:24.633 INFO:teuthology.orchestra.run.smithi031.stdout:(6/6): python3-routes-2.4.1-12.el8.noarch.rpm 294 kB/s | 196 kB 00:00 2023-08-26T13:28:24.636 INFO:teuthology.orchestra.run.smithi031.stdout:-------------------------------------------------------------------------------- 2023-08-26T13:28:24.636 INFO:teuthology.orchestra.run.smithi031.stdout:Total 1.6 MB/s | 2.1 MB 00:01 2023-08-26T13:28:24.649 INFO:teuthology.orchestra.run.smithi031.stdout:Running transaction check 2023-08-26T13:28:24.661 INFO:teuthology.orchestra.run.smithi031.stdout:Transaction check succeeded. 2023-08-26T13:28:24.661 INFO:teuthology.orchestra.run.smithi031.stdout:Running transaction test 2023-08-26T13:28:24.727 INFO:teuthology.orchestra.run.smithi031.stdout:Transaction test succeeded. 2023-08-26T13:28:24.731 INFO:teuthology.orchestra.run.smithi031.stdout:Running transaction 2023-08-26T13:28:24.752 INFO:teuthology.orchestra.run.smithi124.stdout: Installing : python3-repoze-lru-0.7-6.el8.noarch 2/6 2023-08-26T13:28:24.933 INFO:teuthology.orchestra.run.smithi124.stdout: Installing : python3-routes-2.4.1-12.el8.noarch 3/6 2023-08-26T13:28:24.990 INFO:teuthology.orchestra.run.smithi031.stdout: Preparing : 1/1 2023-08-26T13:28:25.103 INFO:teuthology.orchestra.run.smithi124.stdout: Installing : ceph-prometheus-alerts-2:18.0.0-5706.ge7eb9853.el8.n 4/6 2023-08-26T13:28:25.155 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : python3-jwt-1.6.1-2.el8.noarch 1/6 2023-08-26T13:28:25.329 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : python3-repoze-lru-0.7-6.el8.noarch 2/6 2023-08-26T13:28:25.447 INFO:teuthology.orchestra.run.smithi124.stdout: Installing : ceph-grafana-dashboards-2:18.0.0-5706.ge7eb9853.el8. 5/6 2023-08-26T13:28:25.614 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : python3-routes-2.4.1-12.el8.noarch 3/6 2023-08-26T13:28:25.704 INFO:teuthology.orchestra.run.smithi124.stdout: Installing : ceph-mgr-dashboard-2:18.0.0-5706.ge7eb9853.el8.noarc 6/6 2023-08-26T13:28:25.760 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : ceph-prometheus-alerts-2:18.0.0-5706.ge7eb9853.el8.n 4/6 2023-08-26T13:28:25.921 INFO:teuthology.orchestra.run.smithi124.stdout: Running scriptlet: ceph-mgr-dashboard-2:18.0.0-5706.ge7eb9853.el8.noarc 6/6 2023-08-26T13:28:25.921 INFO:teuthology.orchestra.run.smithi124.stdout: Verifying : ceph-grafana-dashboards-2:18.0.0-5706.ge7eb9853.el8. 1/6 2023-08-26T13:28:25.921 INFO:teuthology.orchestra.run.smithi124.stdout: Verifying : ceph-mgr-dashboard-2:18.0.0-5706.ge7eb9853.el8.noarc 2/6 2023-08-26T13:28:25.921 INFO:teuthology.orchestra.run.smithi124.stdout: Verifying : ceph-prometheus-alerts-2:18.0.0-5706.ge7eb9853.el8.n 3/6 2023-08-26T13:28:25.922 INFO:teuthology.orchestra.run.smithi124.stdout: Verifying : python3-repoze-lru-0.7-6.el8.noarch 4/6 2023-08-26T13:28:25.922 INFO:teuthology.orchestra.run.smithi124.stdout: Verifying : python3-routes-2.4.1-12.el8.noarch 5/6 2023-08-26T13:28:26.122 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : ceph-grafana-dashboards-2:18.0.0-5706.ge7eb9853.el8. 5/6 2023-08-26T13:28:26.321 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : ceph-mgr-dashboard-2:18.0.0-5706.ge7eb9853.el8.noarc 6/6 2023-08-26T13:28:26.549 INFO:teuthology.orchestra.run.smithi031.stdout: Running scriptlet: ceph-mgr-dashboard-2:18.0.0-5706.ge7eb9853.el8.noarc 6/6 2023-08-26T13:28:26.549 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : ceph-grafana-dashboards-2:18.0.0-5706.ge7eb9853.el8. 1/6 2023-08-26T13:28:26.549 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : ceph-mgr-dashboard-2:18.0.0-5706.ge7eb9853.el8.noarc 2/6 2023-08-26T13:28:26.550 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : ceph-prometheus-alerts-2:18.0.0-5706.ge7eb9853.el8.n 3/6 2023-08-26T13:28:26.550 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : python3-repoze-lru-0.7-6.el8.noarch 4/6 2023-08-26T13:28:26.550 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : python3-routes-2.4.1-12.el8.noarch 5/6 2023-08-26T13:28:26.981 INFO:teuthology.orchestra.run.smithi124.stdout: Verifying : python3-jwt-1.6.1-2.el8.noarch 6/6 2023-08-26T13:28:26.982 INFO:teuthology.orchestra.run.smithi124.stdout:Last metadata expiration check: 0:02:43 ago on Sat 26 Aug 2023 01:25:43 PM UTC. 2023-08-26T13:28:27.648 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : python3-jwt-1.6.1-2.el8.noarch 6/6 2023-08-26T13:28:27.648 INFO:teuthology.orchestra.run.smithi031.stdout:Last metadata expiration check: 0:02:44 ago on Sat 26 Aug 2023 01:25:43 PM UTC. 2023-08-26T13:28:36.478 INFO:teuthology.orchestra.run.smithi124.stdout:Installed products updated. 2023-08-26T13:28:36.660 INFO:teuthology.orchestra.run.smithi124.stdout: 2023-08-26T13:28:36.661 INFO:teuthology.orchestra.run.smithi124.stdout:Installed: 2023-08-26T13:28:36.661 INFO:teuthology.orchestra.run.smithi124.stdout: ceph-grafana-dashboards-2:18.0.0-5706.ge7eb9853.el8.noarch 2023-08-26T13:28:36.661 INFO:teuthology.orchestra.run.smithi124.stdout: ceph-mgr-dashboard-2:18.0.0-5706.ge7eb9853.el8.noarch 2023-08-26T13:28:36.661 INFO:teuthology.orchestra.run.smithi124.stdout: ceph-prometheus-alerts-2:18.0.0-5706.ge7eb9853.el8.noarch 2023-08-26T13:28:36.661 INFO:teuthology.orchestra.run.smithi124.stdout: python3-jwt-1.6.1-2.el8.noarch 2023-08-26T13:28:36.662 INFO:teuthology.orchestra.run.smithi124.stdout: python3-repoze-lru-0.7-6.el8.noarch 2023-08-26T13:28:36.662 INFO:teuthology.orchestra.run.smithi124.stdout: python3-routes-2.4.1-12.el8.noarch 2023-08-26T13:28:36.662 INFO:teuthology.orchestra.run.smithi124.stdout: 2023-08-26T13:28:36.662 INFO:teuthology.orchestra.run.smithi124.stdout:Complete! 2023-08-26T13:28:36.804 DEBUG:teuthology.orchestra.run.smithi124:> sudo yum -y install ceph-mgr-diskprediction-local 2023-08-26T13:28:37.062 INFO:teuthology.orchestra.run.smithi031.stdout:Installed products updated. 2023-08-26T13:28:37.150 INFO:teuthology.orchestra.run.smithi124.stdout:Updating Subscription Management repositories. 2023-08-26T13:28:37.263 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-08-26T13:28:37.263 INFO:teuthology.orchestra.run.smithi031.stdout:Installed: 2023-08-26T13:28:37.263 INFO:teuthology.orchestra.run.smithi031.stdout: ceph-grafana-dashboards-2:18.0.0-5706.ge7eb9853.el8.noarch 2023-08-26T13:28:37.264 INFO:teuthology.orchestra.run.smithi031.stdout: ceph-mgr-dashboard-2:18.0.0-5706.ge7eb9853.el8.noarch 2023-08-26T13:28:37.264 INFO:teuthology.orchestra.run.smithi031.stdout: ceph-prometheus-alerts-2:18.0.0-5706.ge7eb9853.el8.noarch 2023-08-26T13:28:37.264 INFO:teuthology.orchestra.run.smithi031.stdout: python3-jwt-1.6.1-2.el8.noarch 2023-08-26T13:28:37.264 INFO:teuthology.orchestra.run.smithi031.stdout: python3-repoze-lru-0.7-6.el8.noarch 2023-08-26T13:28:37.264 INFO:teuthology.orchestra.run.smithi031.stdout: python3-routes-2.4.1-12.el8.noarch 2023-08-26T13:28:37.265 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-08-26T13:28:37.265 INFO:teuthology.orchestra.run.smithi031.stdout:Complete! 2023-08-26T13:28:37.419 DEBUG:teuthology.orchestra.run.smithi031:> sudo yum -y install ceph-mgr-diskprediction-local 2023-08-26T13:28:37.789 INFO:teuthology.orchestra.run.smithi031.stdout:Updating Subscription Management repositories. 2023-08-26T13:28:38.686 INFO:teuthology.orchestra.run.smithi124.stdout:Last metadata expiration check: 0:02:55 ago on Sat 26 Aug 2023 01:25:43 PM UTC. 2023-08-26T13:28:39.395 INFO:teuthology.orchestra.run.smithi031.stdout:Last metadata expiration check: 0:02:56 ago on Sat 26 Aug 2023 01:25:43 PM UTC. 2023-08-26T13:28:40.071 INFO:teuthology.orchestra.run.smithi124.stdout:Dependencies resolved. 2023-08-26T13:28:40.072 INFO:teuthology.orchestra.run.smithi124.stdout:======================================================================================================================= 2023-08-26T13:28:40.073 INFO:teuthology.orchestra.run.smithi124.stdout: Package Arch Version Repository Size 2023-08-26T13:28:40.073 INFO:teuthology.orchestra.run.smithi124.stdout:======================================================================================================================= 2023-08-26T13:28:40.073 INFO:teuthology.orchestra.run.smithi124.stdout:Installing: 2023-08-26T13:28:40.073 INFO:teuthology.orchestra.run.smithi124.stdout: ceph-mgr-diskprediction-local noarch 2:18.0.0-5706.ge7eb9853.el8 ceph-noarch 7.4 M 2023-08-26T13:28:40.074 INFO:teuthology.orchestra.run.smithi124.stdout:Installing dependencies: 2023-08-26T13:28:40.074 INFO:teuthology.orchestra.run.smithi124.stdout: libgfortran x86_64 8.5.0-10.1.el8_6 rhel-8-for-x86_64-baseos-rpms 644 k 2023-08-26T13:28:40.074 INFO:teuthology.orchestra.run.smithi124.stdout: libquadmath x86_64 8.5.0-10.1.el8_6 rhel-8-for-x86_64-baseos-rpms 171 k 2023-08-26T13:28:40.074 INFO:teuthology.orchestra.run.smithi124.stdout: openblas x86_64 0.3.15-3.el8 rhel-8-for-x86_64-appstream-rpms 4.7 M 2023-08-26T13:28:40.074 INFO:teuthology.orchestra.run.smithi124.stdout: openblas-threads x86_64 0.3.15-3.el8 rhel-8-for-x86_64-appstream-rpms 4.9 M 2023-08-26T13:28:40.075 INFO:teuthology.orchestra.run.smithi124.stdout: python3-numpy x86_64 1:1.14.3-10.el8 rhel-8-for-x86_64-appstream-rpms 3.7 M 2023-08-26T13:28:40.075 INFO:teuthology.orchestra.run.smithi124.stdout: python3-numpy-f2py x86_64 1:1.14.3-10.el8 rhel-8-for-x86_64-appstream-rpms 225 k 2023-08-26T13:28:40.075 INFO:teuthology.orchestra.run.smithi124.stdout: python3-scipy x86_64 1.0.0-21.module+el8.5.0+10916+41bd434d rhel-8-for-x86_64-appstream-rpms 14 M 2023-08-26T13:28:40.075 INFO:teuthology.orchestra.run.smithi124.stdout: 2023-08-26T13:28:40.076 INFO:teuthology.orchestra.run.smithi124.stdout:Transaction Summary 2023-08-26T13:28:40.076 INFO:teuthology.orchestra.run.smithi124.stdout:======================================================================================================================= 2023-08-26T13:28:40.076 INFO:teuthology.orchestra.run.smithi124.stdout:Install 8 Packages 2023-08-26T13:28:40.076 INFO:teuthology.orchestra.run.smithi124.stdout: 2023-08-26T13:28:40.077 INFO:teuthology.orchestra.run.smithi124.stdout:Total download size: 35 M 2023-08-26T13:28:40.077 INFO:teuthology.orchestra.run.smithi124.stdout:Installed size: 210 M 2023-08-26T13:28:40.077 INFO:teuthology.orchestra.run.smithi124.stdout:Downloading Packages: 2023-08-26T13:28:40.272 INFO:teuthology.orchestra.run.smithi124.stdout:(1/8): libquadmath-8.5.0-10.1.el8_6.x86_64.rpm 868 kB/s | 171 kB 00:00 2023-08-26T13:28:40.306 INFO:teuthology.orchestra.run.smithi124.stdout:(2/8): libgfortran-8.5.0-10.1.el8_6.x86_64.rpm 2.7 MB/s | 644 kB 00:00 2023-08-26T13:28:40.515 INFO:teuthology.orchestra.run.smithi124.stdout:(3/8): python3-numpy-1.14.3-10.el8.x86_64.rpm 18 MB/s | 3.7 MB 00:00 2023-08-26T13:28:40.740 INFO:teuthology.orchestra.run.smithi124.stdout:(4/8): python3-scipy-1.0.0-21.module+el8.5.0+10 29 MB/s | 14 MB 00:00 2023-08-26T13:28:40.766 INFO:teuthology.orchestra.run.smithi124.stdout:(5/8): python3-numpy-f2py-1.14.3-10.el8.x86_64. 899 kB/s | 225 kB 00:00 2023-08-26T13:28:40.944 INFO:teuthology.orchestra.run.smithi031.stdout:Dependencies resolved. 2023-08-26T13:28:40.946 INFO:teuthology.orchestra.run.smithi031.stdout:======================================================================================================================= 2023-08-26T13:28:40.946 INFO:teuthology.orchestra.run.smithi031.stdout: Package Arch Version Repository Size 2023-08-26T13:28:40.946 INFO:teuthology.orchestra.run.smithi031.stdout:======================================================================================================================= 2023-08-26T13:28:40.946 INFO:teuthology.orchestra.run.smithi031.stdout:Installing: 2023-08-26T13:28:40.947 INFO:teuthology.orchestra.run.smithi031.stdout: ceph-mgr-diskprediction-local noarch 2:18.0.0-5706.ge7eb9853.el8 ceph-noarch 7.4 M 2023-08-26T13:28:40.947 INFO:teuthology.orchestra.run.smithi031.stdout:Installing dependencies: 2023-08-26T13:28:40.947 INFO:teuthology.orchestra.run.smithi031.stdout: libgfortran x86_64 8.5.0-10.1.el8_6 rhel-8-for-x86_64-baseos-rpms 644 k 2023-08-26T13:28:40.947 INFO:teuthology.orchestra.run.smithi031.stdout: libquadmath x86_64 8.5.0-10.1.el8_6 rhel-8-for-x86_64-baseos-rpms 171 k 2023-08-26T13:28:40.947 INFO:teuthology.orchestra.run.smithi031.stdout: openblas x86_64 0.3.15-3.el8 rhel-8-for-x86_64-appstream-rpms 4.7 M 2023-08-26T13:28:40.948 INFO:teuthology.orchestra.run.smithi031.stdout: openblas-threads x86_64 0.3.15-3.el8 rhel-8-for-x86_64-appstream-rpms 4.9 M 2023-08-26T13:28:40.948 INFO:teuthology.orchestra.run.smithi031.stdout: python3-numpy x86_64 1:1.14.3-10.el8 rhel-8-for-x86_64-appstream-rpms 3.7 M 2023-08-26T13:28:40.948 INFO:teuthology.orchestra.run.smithi031.stdout: python3-numpy-f2py x86_64 1:1.14.3-10.el8 rhel-8-for-x86_64-appstream-rpms 225 k 2023-08-26T13:28:40.948 INFO:teuthology.orchestra.run.smithi031.stdout: python3-scipy x86_64 1.0.0-21.module+el8.5.0+10916+41bd434d rhel-8-for-x86_64-appstream-rpms 14 M 2023-08-26T13:28:40.948 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-08-26T13:28:40.949 INFO:teuthology.orchestra.run.smithi031.stdout:Transaction Summary 2023-08-26T13:28:40.949 INFO:teuthology.orchestra.run.smithi031.stdout:======================================================================================================================= 2023-08-26T13:28:40.949 INFO:teuthology.orchestra.run.smithi031.stdout:Install 8 Packages 2023-08-26T13:28:40.949 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-08-26T13:28:40.950 INFO:teuthology.orchestra.run.smithi031.stdout:Total download size: 35 M 2023-08-26T13:28:40.950 INFO:teuthology.orchestra.run.smithi031.stdout:Installed size: 210 M 2023-08-26T13:28:40.950 INFO:teuthology.orchestra.run.smithi031.stdout:Downloading Packages: 2023-08-26T13:28:40.958 INFO:teuthology.orchestra.run.smithi124.stdout:(6/8): openblas-threads-0.3.15-3.el8.x86_64.rpm 22 MB/s | 4.9 MB 00:00 2023-08-26T13:28:41.049 INFO:teuthology.orchestra.run.smithi124.stdout:(7/8): openblas-0.3.15-3.el8.x86_64.rpm 17 MB/s | 4.7 MB 00:00 2023-08-26T13:28:41.114 INFO:teuthology.orchestra.run.smithi031.stdout:(1/8): libquadmath-8.5.0-10.1.el8_6.x86_64.rpm 1.0 MB/s | 171 kB 00:00 2023-08-26T13:28:41.139 INFO:teuthology.orchestra.run.smithi031.stdout:(2/8): libgfortran-8.5.0-10.1.el8_6.x86_64.rpm 3.3 MB/s | 644 kB 00:00 2023-08-26T13:28:41.366 INFO:teuthology.orchestra.run.smithi124.stdout:(8/8): ceph-mgr-diskprediction-local-18.0.0-570 5.7 MB/s | 7.4 MB 00:01 2023-08-26T13:28:41.367 INFO:teuthology.orchestra.run.smithi124.stdout:-------------------------------------------------------------------------------- 2023-08-26T13:28:41.368 INFO:teuthology.orchestra.run.smithi124.stdout:Total 27 MB/s | 35 MB 00:01 2023-08-26T13:28:41.372 INFO:teuthology.orchestra.run.smithi031.stdout:(3/8): python3-numpy-1.14.3-10.el8.x86_64.rpm 16 MB/s | 3.7 MB 00:00 2023-08-26T13:28:41.489 INFO:teuthology.orchestra.run.smithi031.stdout:(4/8): python3-numpy-f2py-1.14.3-10.el8.x86_64. 1.9 MB/s | 225 kB 00:00 2023-08-26T13:28:41.606 INFO:teuthology.orchestra.run.smithi124.stdout:Running transaction check 2023-08-26T13:28:41.624 INFO:teuthology.orchestra.run.smithi124.stdout:Transaction check succeeded. 2023-08-26T13:28:41.624 INFO:teuthology.orchestra.run.smithi124.stdout:Running transaction test 2023-08-26T13:28:41.640 INFO:teuthology.orchestra.run.smithi031.stdout:(5/8): python3-scipy-1.0.0-21.module+el8.5.0+10 26 MB/s | 14 MB 00:00 2023-08-26T13:28:41.765 INFO:teuthology.orchestra.run.smithi031.stdout:(6/8): openblas-threads-0.3.15-3.el8.x86_64.rpm 18 MB/s | 4.9 MB 00:00 2023-08-26T13:28:41.915 INFO:teuthology.orchestra.run.smithi124.stdout:Transaction test succeeded. 2023-08-26T13:28:41.919 INFO:teuthology.orchestra.run.smithi124.stdout:Running transaction 2023-08-26T13:28:42.007 INFO:teuthology.orchestra.run.smithi031.stdout:(7/8): openblas-0.3.15-3.el8.x86_64.rpm 13 MB/s | 4.7 MB 00:00 2023-08-26T13:28:42.332 INFO:teuthology.orchestra.run.smithi031.stdout:(8/8): ceph-mgr-diskprediction-local-18.0.0-570 5.4 MB/s | 7.4 MB 00:01 2023-08-26T13:28:42.333 INFO:teuthology.orchestra.run.smithi031.stdout:-------------------------------------------------------------------------------- 2023-08-26T13:28:42.333 INFO:teuthology.orchestra.run.smithi031.stdout:Total 26 MB/s | 35 MB 00:01 2023-08-26T13:28:42.471 INFO:teuthology.orchestra.run.smithi124.stdout: Preparing : 1/1 2023-08-26T13:28:42.576 INFO:teuthology.orchestra.run.smithi031.stdout:Running transaction check 2023-08-26T13:28:42.596 INFO:teuthology.orchestra.run.smithi031.stdout:Transaction check succeeded. 2023-08-26T13:28:42.596 INFO:teuthology.orchestra.run.smithi031.stdout:Running transaction test 2023-08-26T13:28:42.652 INFO:teuthology.orchestra.run.smithi124.stdout: Installing : libquadmath-8.5.0-10.1.el8_6.x86_64 1/8 2023-08-26T13:28:42.799 INFO:teuthology.orchestra.run.smithi124.stdout: Running scriptlet: libquadmath-8.5.0-10.1.el8_6.x86_64 1/8 2023-08-26T13:28:42.905 INFO:teuthology.orchestra.run.smithi031.stdout:Transaction test succeeded. 2023-08-26T13:28:42.909 INFO:teuthology.orchestra.run.smithi031.stdout:Running transaction 2023-08-26T13:28:42.952 INFO:teuthology.orchestra.run.smithi124.stdout: Installing : libgfortran-8.5.0-10.1.el8_6.x86_64 2/8 2023-08-26T13:28:43.492 INFO:teuthology.orchestra.run.smithi031.stdout: Preparing : 1/1 2023-08-26T13:28:43.587 INFO:teuthology.orchestra.run.smithi124.stdout: Running scriptlet: libgfortran-8.5.0-10.1.el8_6.x86_64 2/8 2023-08-26T13:28:43.624 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : libquadmath-8.5.0-10.1.el8_6.x86_64 1/8 2023-08-26T13:28:43.750 INFO:teuthology.orchestra.run.smithi031.stdout: Running scriptlet: libquadmath-8.5.0-10.1.el8_6.x86_64 1/8 2023-08-26T13:28:43.761 INFO:teuthology.orchestra.run.smithi124.stdout: Installing : openblas-threads-0.3.15-3.el8.x86_64 3/8 2023-08-26T13:28:43.874 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : libgfortran-8.5.0-10.1.el8_6.x86_64 2/8 2023-08-26T13:28:44.340 INFO:teuthology.orchestra.run.smithi124.stdout: Running scriptlet: openblas-threads-0.3.15-3.el8.x86_64 3/8 2023-08-26T13:28:44.494 INFO:teuthology.orchestra.run.smithi124.stdout: Installing : openblas-0.3.15-3.el8.x86_64 4/8 2023-08-26T13:28:44.530 INFO:teuthology.orchestra.run.smithi031.stdout: Running scriptlet: libgfortran-8.5.0-10.1.el8_6.x86_64 2/8 2023-08-26T13:28:44.683 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : openblas-threads-0.3.15-3.el8.x86_64 3/8 2023-08-26T13:28:45.174 INFO:teuthology.orchestra.run.smithi124.stdout: Running scriptlet: openblas-0.3.15-3.el8.x86_64 4/8 2023-08-26T13:28:45.318 INFO:teuthology.orchestra.run.smithi031.stdout: Running scriptlet: openblas-threads-0.3.15-3.el8.x86_64 3/8 2023-08-26T13:28:45.466 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : openblas-0.3.15-3.el8.x86_64 4/8 2023-08-26T13:28:45.513 INFO:teuthology.orchestra.run.smithi124.stdout: Installing : python3-numpy-1:1.14.3-10.el8.x86_64 5/8 2023-08-26T13:28:45.702 INFO:teuthology.orchestra.run.smithi124.stdout: Installing : python3-numpy-f2py-1:1.14.3-10.el8.x86_64 6/8 2023-08-26T13:28:46.203 INFO:teuthology.orchestra.run.smithi031.stdout: Running scriptlet: openblas-0.3.15-3.el8.x86_64 4/8 2023-08-26T13:28:46.491 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : python3-numpy-1:1.14.3-10.el8.x86_64 5/8 2023-08-26T13:28:46.666 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : python3-numpy-f2py-1:1.14.3-10.el8.x86_64 6/8 2023-08-26T13:28:47.459 INFO:teuthology.orchestra.run.smithi124.stdout: Running scriptlet: python3-numpy-f2py-1:1.14.3-10.el8.x86_64 6/8 2023-08-26T13:28:48.532 INFO:teuthology.orchestra.run.smithi031.stdout: Running scriptlet: python3-numpy-f2py-1:1.14.3-10.el8.x86_64 6/8 2023-08-26T13:28:48.714 INFO:teuthology.orchestra.run.smithi124.stdout: Installing : python3-scipy-1.0.0-21.module+el8.5.0+10916+41bd434d 7/8 2023-08-26T13:28:48.902 INFO:teuthology.orchestra.run.smithi124.stdout: Installing : ceph-mgr-diskprediction-local-2:18.0.0-5706.ge7eb985 8/8 2023-08-26T13:28:49.150 INFO:teuthology.orchestra.run.smithi124.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:18.0.0-5706.ge7eb985 8/8 2023-08-26T13:28:49.151 INFO:teuthology.orchestra.run.smithi124.stdout: Verifying : ceph-mgr-diskprediction-local-2:18.0.0-5706.ge7eb985 1/8 2023-08-26T13:28:49.151 INFO:teuthology.orchestra.run.smithi124.stdout: Verifying : libgfortran-8.5.0-10.1.el8_6.x86_64 2/8 2023-08-26T13:28:49.151 INFO:teuthology.orchestra.run.smithi124.stdout: Verifying : libquadmath-8.5.0-10.1.el8_6.x86_64 3/8 2023-08-26T13:28:49.151 INFO:teuthology.orchestra.run.smithi124.stdout: Verifying : python3-scipy-1.0.0-21.module+el8.5.0+10916+41bd434d 4/8 2023-08-26T13:28:49.151 INFO:teuthology.orchestra.run.smithi124.stdout: Verifying : python3-numpy-1:1.14.3-10.el8.x86_64 5/8 2023-08-26T13:28:49.152 INFO:teuthology.orchestra.run.smithi124.stdout: Verifying : python3-numpy-f2py-1:1.14.3-10.el8.x86_64 6/8 2023-08-26T13:28:49.152 INFO:teuthology.orchestra.run.smithi124.stdout: Verifying : openblas-threads-0.3.15-3.el8.x86_64 7/8 2023-08-26T13:28:49.810 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : python3-scipy-1.0.0-21.module+el8.5.0+10916+41bd434d 7/8 2023-08-26T13:28:49.932 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : ceph-mgr-diskprediction-local-2:18.0.0-5706.ge7eb985 8/8 2023-08-26T13:28:50.182 INFO:teuthology.orchestra.run.smithi031.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:18.0.0-5706.ge7eb985 8/8 2023-08-26T13:28:50.182 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : ceph-mgr-diskprediction-local-2:18.0.0-5706.ge7eb985 1/8 2023-08-26T13:28:50.182 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : libgfortran-8.5.0-10.1.el8_6.x86_64 2/8 2023-08-26T13:28:50.183 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : libquadmath-8.5.0-10.1.el8_6.x86_64 3/8 2023-08-26T13:28:50.183 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : python3-scipy-1.0.0-21.module+el8.5.0+10916+41bd434d 4/8 2023-08-26T13:28:50.183 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : python3-numpy-1:1.14.3-10.el8.x86_64 5/8 2023-08-26T13:28:50.183 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : python3-numpy-f2py-1:1.14.3-10.el8.x86_64 6/8 2023-08-26T13:28:50.183 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : openblas-threads-0.3.15-3.el8.x86_64 7/8 2023-08-26T13:28:50.249 INFO:teuthology.orchestra.run.smithi124.stdout: Verifying : openblas-0.3.15-3.el8.x86_64 8/8 2023-08-26T13:28:50.249 INFO:teuthology.orchestra.run.smithi124.stdout:Last metadata expiration check: 0:03:07 ago on Sat 26 Aug 2023 01:25:43 PM UTC. 2023-08-26T13:28:51.263 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : openblas-0.3.15-3.el8.x86_64 8/8 2023-08-26T13:28:51.264 INFO:teuthology.orchestra.run.smithi031.stdout:Last metadata expiration check: 0:03:08 ago on Sat 26 Aug 2023 01:25:43 PM UTC. 2023-08-26T13:28:57.117 INFO:teuthology.orchestra.run.smithi124.stdout:Installed products updated. 2023-08-26T13:28:57.299 INFO:teuthology.orchestra.run.smithi124.stdout: 2023-08-26T13:28:57.300 INFO:teuthology.orchestra.run.smithi124.stdout:Installed: 2023-08-26T13:28:57.300 INFO:teuthology.orchestra.run.smithi124.stdout: ceph-mgr-diskprediction-local-2:18.0.0-5706.ge7eb9853.el8.noarch 2023-08-26T13:28:57.300 INFO:teuthology.orchestra.run.smithi124.stdout: libgfortran-8.5.0-10.1.el8_6.x86_64 2023-08-26T13:28:57.300 INFO:teuthology.orchestra.run.smithi124.stdout: libquadmath-8.5.0-10.1.el8_6.x86_64 2023-08-26T13:28:57.301 INFO:teuthology.orchestra.run.smithi124.stdout: openblas-0.3.15-3.el8.x86_64 2023-08-26T13:28:57.301 INFO:teuthology.orchestra.run.smithi124.stdout: openblas-threads-0.3.15-3.el8.x86_64 2023-08-26T13:28:57.301 INFO:teuthology.orchestra.run.smithi124.stdout: python3-numpy-1:1.14.3-10.el8.x86_64 2023-08-26T13:28:57.301 INFO:teuthology.orchestra.run.smithi124.stdout: python3-numpy-f2py-1:1.14.3-10.el8.x86_64 2023-08-26T13:28:57.301 INFO:teuthology.orchestra.run.smithi124.stdout: python3-scipy-1.0.0-21.module+el8.5.0+10916+41bd434d.x86_64 2023-08-26T13:28:57.302 INFO:teuthology.orchestra.run.smithi124.stdout: 2023-08-26T13:28:57.302 INFO:teuthology.orchestra.run.smithi124.stdout:Complete! 2023-08-26T13:28:57.425 DEBUG:teuthology.orchestra.run.smithi124:> sudo yum -y install ceph-mgr-rook 2023-08-26T13:28:57.775 INFO:teuthology.orchestra.run.smithi124.stdout:Updating Subscription Management repositories. 2023-08-26T13:28:59.251 INFO:teuthology.orchestra.run.smithi124.stdout:Last metadata expiration check: 0:03:16 ago on Sat 26 Aug 2023 01:25:43 PM UTC. 2023-08-26T13:29:00.328 INFO:teuthology.orchestra.run.smithi031.stdout:Installed products updated. 2023-08-26T13:29:00.533 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-08-26T13:29:00.534 INFO:teuthology.orchestra.run.smithi031.stdout:Installed: 2023-08-26T13:29:00.534 INFO:teuthology.orchestra.run.smithi031.stdout: ceph-mgr-diskprediction-local-2:18.0.0-5706.ge7eb9853.el8.noarch 2023-08-26T13:29:00.534 INFO:teuthology.orchestra.run.smithi031.stdout: libgfortran-8.5.0-10.1.el8_6.x86_64 2023-08-26T13:29:00.534 INFO:teuthology.orchestra.run.smithi031.stdout: libquadmath-8.5.0-10.1.el8_6.x86_64 2023-08-26T13:29:00.535 INFO:teuthology.orchestra.run.smithi031.stdout: openblas-0.3.15-3.el8.x86_64 2023-08-26T13:29:00.535 INFO:teuthology.orchestra.run.smithi031.stdout: openblas-threads-0.3.15-3.el8.x86_64 2023-08-26T13:29:00.535 INFO:teuthology.orchestra.run.smithi031.stdout: python3-numpy-1:1.14.3-10.el8.x86_64 2023-08-26T13:29:00.535 INFO:teuthology.orchestra.run.smithi031.stdout: python3-numpy-f2py-1:1.14.3-10.el8.x86_64 2023-08-26T13:29:00.535 INFO:teuthology.orchestra.run.smithi031.stdout: python3-scipy-1.0.0-21.module+el8.5.0+10916+41bd434d.x86_64 2023-08-26T13:29:00.535 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-08-26T13:29:00.536 INFO:teuthology.orchestra.run.smithi031.stdout:Complete! 2023-08-26T13:29:00.625 INFO:teuthology.orchestra.run.smithi124.stdout:Dependencies resolved. 2023-08-26T13:29:00.626 INFO:teuthology.orchestra.run.smithi124.stdout:======================================================================================================== 2023-08-26T13:29:00.626 INFO:teuthology.orchestra.run.smithi124.stdout: Package Arch Version Repository Size 2023-08-26T13:29:00.627 INFO:teuthology.orchestra.run.smithi124.stdout:======================================================================================================== 2023-08-26T13:29:00.627 INFO:teuthology.orchestra.run.smithi124.stdout:Installing: 2023-08-26T13:29:00.627 INFO:teuthology.orchestra.run.smithi124.stdout: ceph-mgr-rook noarch 2:18.0.0-5706.ge7eb9853.el8 ceph-noarch 47 k 2023-08-26T13:29:00.627 INFO:teuthology.orchestra.run.smithi124.stdout:Installing dependencies: 2023-08-26T13:29:00.627 INFO:teuthology.orchestra.run.smithi124.stdout: python3-cachetools noarch 3.1.1-4.el8 epel 33 k 2023-08-26T13:29:00.628 INFO:teuthology.orchestra.run.smithi124.stdout: python3-certifi noarch 2018.10.15-7.el8 epel 16 k 2023-08-26T13:29:00.628 INFO:teuthology.orchestra.run.smithi124.stdout: python3-google-auth noarch 1:1.1.1-10.el8 epel 81 k 2023-08-26T13:29:00.628 INFO:teuthology.orchestra.run.smithi124.stdout: python3-jsonpatch noarch 1.21-2.el8 rhel-8-for-x86_64-appstream-rpms 27 k 2023-08-26T13:29:00.628 INFO:teuthology.orchestra.run.smithi124.stdout: python3-jsonpointer noarch 1.10-11.el8 rhel-8-for-x86_64-appstream-rpms 20 k 2023-08-26T13:29:00.628 INFO:teuthology.orchestra.run.smithi124.stdout: python3-kubernetes noarch 1:11.0.0-6.el8 epel 1.2 M 2023-08-26T13:29:00.629 INFO:teuthology.orchestra.run.smithi124.stdout: python3-oauthlib noarch 2.1.0-1.el8 rhel-8-for-x86_64-baseos-rpms 155 k 2023-08-26T13:29:00.629 INFO:teuthology.orchestra.run.smithi124.stdout: python3-pyasn1 noarch 0.3.7-6.el8 rhel-8-for-x86_64-appstream-rpms 126 k 2023-08-26T13:29:00.629 INFO:teuthology.orchestra.run.smithi124.stdout: python3-pyasn1-modules noarch 0.3.7-6.el8 rhel-8-for-x86_64-appstream-rpms 110 k 2023-08-26T13:29:00.629 INFO:teuthology.orchestra.run.smithi124.stdout: python3-requests-oauthlib noarch 1.0.0-1.el8 rhel-8-for-x86_64-baseos-rpms 43 k 2023-08-26T13:29:00.629 INFO:teuthology.orchestra.run.smithi124.stdout: python3-rsa noarch 4.9-2.el8 epel 64 k 2023-08-26T13:29:00.630 INFO:teuthology.orchestra.run.smithi124.stdout: python3-websocket-client noarch 0.56.0-5.el8 epel 61 k 2023-08-26T13:29:00.630 INFO:teuthology.orchestra.run.smithi124.stdout: 2023-08-26T13:29:00.630 INFO:teuthology.orchestra.run.smithi124.stdout:Transaction Summary 2023-08-26T13:29:00.630 INFO:teuthology.orchestra.run.smithi124.stdout:======================================================================================================== 2023-08-26T13:29:00.630 INFO:teuthology.orchestra.run.smithi124.stdout:Install 13 Packages 2023-08-26T13:29:00.631 INFO:teuthology.orchestra.run.smithi124.stdout: 2023-08-26T13:29:00.631 INFO:teuthology.orchestra.run.smithi124.stdout:Total download size: 1.9 M 2023-08-26T13:29:00.632 INFO:teuthology.orchestra.run.smithi124.stdout:Installed size: 25 M 2023-08-26T13:29:00.632 INFO:teuthology.orchestra.run.smithi124.stdout:Downloading Packages: 2023-08-26T13:29:00.723 DEBUG:teuthology.orchestra.run.smithi031:> sudo yum -y install ceph-mgr-rook 2023-08-26T13:29:00.945 INFO:teuthology.orchestra.run.smithi124.stdout:(1/13): python3-certifi-2018.10.15-7.el8.noarch 112 kB/s | 16 kB 00:00 2023-08-26T13:29:00.970 INFO:teuthology.orchestra.run.smithi124.stdout:(2/13): ceph-mgr-rook-18.0.0-5706.ge7eb9853.el8 277 kB/s | 47 kB 00:00 2023-08-26T13:29:01.029 INFO:teuthology.orchestra.run.smithi124.stdout:(3/13): python3-cachetools-3.1.1-4.el8.noarch.r 144 kB/s | 33 kB 00:00 2023-08-26T13:29:01.080 INFO:teuthology.orchestra.run.smithi124.stdout:(4/13): python3-google-auth-1.1.1-10.el8.noarch 606 kB/s | 81 kB 00:00 2023-08-26T13:29:01.089 INFO:teuthology.orchestra.run.smithi031.stdout:Updating Subscription Management repositories. 2023-08-26T13:29:01.163 INFO:teuthology.orchestra.run.smithi124.stdout:(5/13): python3-websocket-client-0.56.0-5.el8.n 734 kB/s | 61 kB 00:00 2023-08-26T13:29:01.231 INFO:teuthology.orchestra.run.smithi124.stdout:(6/13): python3-rsa-4.9-2.el8.noarch.rpm 319 kB/s | 64 kB 00:00 2023-08-26T13:29:01.331 INFO:teuthology.orchestra.run.smithi124.stdout:(7/13): python3-kubernetes-11.0.0-6.el8.noarch. 3.2 MB/s | 1.2 MB 00:00 2023-08-26T13:29:01.356 INFO:teuthology.orchestra.run.smithi124.stdout:(8/13): python3-requests-oauthlib-1.0.0-1.el8.n 222 kB/s | 43 kB 00:00 2023-08-26T13:29:01.398 INFO:teuthology.orchestra.run.smithi124.stdout:(9/13): python3-oauthlib-2.1.0-1.el8.noarch.rpm 926 kB/s | 155 kB 00:00 2023-08-26T13:29:01.473 INFO:teuthology.orchestra.run.smithi124.stdout:(10/13): python3-pyasn1-modules-0.3.7-6.el8.noa 773 kB/s | 110 kB 00:00 2023-08-26T13:29:01.499 INFO:teuthology.orchestra.run.smithi124.stdout:(11/13): python3-jsonpatch-1.21-2.el8.noarch.rp 194 kB/s | 27 kB 00:00 2023-08-26T13:29:01.616 INFO:teuthology.orchestra.run.smithi124.stdout:(12/13): python3-pyasn1-0.3.7-6.el8.noarch.rpm 582 kB/s | 126 kB 00:00 2023-08-26T13:29:01.641 INFO:teuthology.orchestra.run.smithi124.stdout:(13/13): python3-jsonpointer-1.10-11.el8.noarch 120 kB/s | 20 kB 00:00 2023-08-26T13:29:01.641 INFO:teuthology.orchestra.run.smithi124.stdout:-------------------------------------------------------------------------------- 2023-08-26T13:29:01.642 INFO:teuthology.orchestra.run.smithi124.stdout:Total 1.9 MB/s | 1.9 MB 00:01 2023-08-26T13:29:01.708 INFO:teuthology.orchestra.run.smithi124.stdout:Running transaction check 2023-08-26T13:29:01.724 INFO:teuthology.orchestra.run.smithi124.stdout:Transaction check succeeded. 2023-08-26T13:29:01.724 INFO:teuthology.orchestra.run.smithi124.stdout:Running transaction test 2023-08-26T13:29:01.795 INFO:teuthology.orchestra.run.smithi124.stdout:Transaction test succeeded. 2023-08-26T13:29:01.799 INFO:teuthology.orchestra.run.smithi124.stdout:Running transaction 2023-08-26T13:29:02.295 INFO:teuthology.orchestra.run.smithi124.stdout: Preparing : 1/1 2023-08-26T13:29:02.483 INFO:teuthology.orchestra.run.smithi124.stdout: Installing : python3-pyasn1-0.3.7-6.el8.noarch 1/13 2023-08-26T13:29:02.702 INFO:teuthology.orchestra.run.smithi124.stdout: Installing : python3-rsa-4.9-2.el8.noarch 2/13 2023-08-26T13:29:02.755 INFO:teuthology.orchestra.run.smithi031.stdout:Last metadata expiration check: 0:03:19 ago on Sat 26 Aug 2023 01:25:43 PM UTC. 2023-08-26T13:29:02.898 INFO:teuthology.orchestra.run.smithi124.stdout: Installing : python3-pyasn1-modules-0.3.7-6.el8.noarch 3/13 2023-08-26T13:29:03.049 INFO:teuthology.orchestra.run.smithi124.stdout: Installing : python3-jsonpointer-1.10-11.el8.noarch 4/13 2023-08-26T13:29:03.245 INFO:teuthology.orchestra.run.smithi124.stdout: Installing : python3-jsonpatch-1.21-2.el8.noarch 5/13 2023-08-26T13:29:03.496 INFO:teuthology.orchestra.run.smithi124.stdout: Installing : python3-oauthlib-2.1.0-1.el8.noarch 6/13 2023-08-26T13:29:03.694 INFO:teuthology.orchestra.run.smithi124.stdout: Installing : python3-requests-oauthlib-1.0.0-1.el8.noarch 7/13 2023-08-26T13:29:03.874 INFO:teuthology.orchestra.run.smithi124.stdout: Installing : python3-websocket-client-0.56.0-5.el8.noarch 8/13 2023-08-26T13:29:04.056 INFO:teuthology.orchestra.run.smithi124.stdout: Installing : python3-certifi-2018.10.15-7.el8.noarch 9/13 2023-08-26T13:29:04.255 INFO:teuthology.orchestra.run.smithi124.stdout: Installing : python3-cachetools-3.1.1-4.el8.noarch 10/13 2023-08-26T13:29:04.268 INFO:teuthology.orchestra.run.smithi031.stdout:Dependencies resolved. 2023-08-26T13:29:04.270 INFO:teuthology.orchestra.run.smithi031.stdout:======================================================================================================== 2023-08-26T13:29:04.271 INFO:teuthology.orchestra.run.smithi031.stdout: Package Arch Version Repository Size 2023-08-26T13:29:04.271 INFO:teuthology.orchestra.run.smithi031.stdout:======================================================================================================== 2023-08-26T13:29:04.271 INFO:teuthology.orchestra.run.smithi031.stdout:Installing: 2023-08-26T13:29:04.272 INFO:teuthology.orchestra.run.smithi031.stdout: ceph-mgr-rook noarch 2:18.0.0-5706.ge7eb9853.el8 ceph-noarch 47 k 2023-08-26T13:29:04.272 INFO:teuthology.orchestra.run.smithi031.stdout:Installing dependencies: 2023-08-26T13:29:04.272 INFO:teuthology.orchestra.run.smithi031.stdout: python3-cachetools noarch 3.1.1-4.el8 epel 33 k 2023-08-26T13:29:04.272 INFO:teuthology.orchestra.run.smithi031.stdout: python3-certifi noarch 2018.10.15-7.el8 epel 16 k 2023-08-26T13:29:04.272 INFO:teuthology.orchestra.run.smithi031.stdout: python3-google-auth noarch 1:1.1.1-10.el8 epel 81 k 2023-08-26T13:29:04.273 INFO:teuthology.orchestra.run.smithi031.stdout: python3-jsonpatch noarch 1.21-2.el8 rhel-8-for-x86_64-appstream-rpms 27 k 2023-08-26T13:29:04.273 INFO:teuthology.orchestra.run.smithi031.stdout: python3-jsonpointer noarch 1.10-11.el8 rhel-8-for-x86_64-appstream-rpms 20 k 2023-08-26T13:29:04.273 INFO:teuthology.orchestra.run.smithi031.stdout: python3-kubernetes noarch 1:11.0.0-6.el8 epel 1.2 M 2023-08-26T13:29:04.273 INFO:teuthology.orchestra.run.smithi031.stdout: python3-oauthlib noarch 2.1.0-1.el8 rhel-8-for-x86_64-baseos-rpms 155 k 2023-08-26T13:29:04.273 INFO:teuthology.orchestra.run.smithi031.stdout: python3-pyasn1 noarch 0.3.7-6.el8 rhel-8-for-x86_64-appstream-rpms 126 k 2023-08-26T13:29:04.274 INFO:teuthology.orchestra.run.smithi031.stdout: python3-pyasn1-modules noarch 0.3.7-6.el8 rhel-8-for-x86_64-appstream-rpms 110 k 2023-08-26T13:29:04.274 INFO:teuthology.orchestra.run.smithi031.stdout: python3-requests-oauthlib noarch 1.0.0-1.el8 rhel-8-for-x86_64-baseos-rpms 43 k 2023-08-26T13:29:04.274 INFO:teuthology.orchestra.run.smithi031.stdout: python3-rsa noarch 4.9-2.el8 epel 64 k 2023-08-26T13:29:04.274 INFO:teuthology.orchestra.run.smithi031.stdout: python3-websocket-client noarch 0.56.0-5.el8 epel 61 k 2023-08-26T13:29:04.275 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-08-26T13:29:04.275 INFO:teuthology.orchestra.run.smithi031.stdout:Transaction Summary 2023-08-26T13:29:04.275 INFO:teuthology.orchestra.run.smithi031.stdout:======================================================================================================== 2023-08-26T13:29:04.275 INFO:teuthology.orchestra.run.smithi031.stdout:Install 13 Packages 2023-08-26T13:29:04.275 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-08-26T13:29:04.276 INFO:teuthology.orchestra.run.smithi031.stdout:Total download size: 1.9 M 2023-08-26T13:29:04.276 INFO:teuthology.orchestra.run.smithi031.stdout:Installed size: 25 M 2023-08-26T13:29:04.277 INFO:teuthology.orchestra.run.smithi031.stdout:Downloading Packages: 2023-08-26T13:29:04.944 INFO:teuthology.orchestra.run.smithi031.stdout:(1/13): python3-certifi-2018.10.15-7.el8.noarch 62 kB/s | 16 kB 00:00 2023-08-26T13:29:04.961 INFO:teuthology.orchestra.run.smithi031.stdout:(2/13): python3-cachetools-3.1.1-4.el8.noarch.r 118 kB/s | 33 kB 00:00 2023-08-26T13:29:05.162 INFO:teuthology.orchestra.run.smithi031.stdout:(3/13): ceph-mgr-rook-18.0.0-5706.ge7eb9853.el8 99 kB/s | 47 kB 00:00 2023-08-26T13:29:05.187 INFO:teuthology.orchestra.run.smithi031.stdout:(4/13): python3-google-auth-1.1.1-10.el8.noarch 335 kB/s | 81 kB 00:00 2023-08-26T13:29:05.295 INFO:teuthology.orchestra.run.smithi031.stdout:(5/13): python3-websocket-client-0.56.0-5.el8.n 567 kB/s | 61 kB 00:00 2023-08-26T13:29:05.345 INFO:teuthology.orchestra.run.smithi124.stdout: Installing : python3-google-auth-1:1.1.1-10.el8.noarch 11/13 2023-08-26T13:29:05.429 INFO:teuthology.orchestra.run.smithi031.stdout:(6/13): python3-requests-oauthlib-1.0.0-1.el8.n 320 kB/s | 43 kB 00:00 2023-08-26T13:29:05.454 INFO:teuthology.orchestra.run.smithi031.stdout:(7/13): python3-rsa-4.9-2.el8.noarch.rpm 220 kB/s | 64 kB 00:00 2023-08-26T13:29:05.496 INFO:teuthology.orchestra.run.smithi031.stdout:(8/13): python3-kubernetes-11.0.0-6.el8.noarch. 2.2 MB/s | 1.2 MB 00:00 2023-08-26T13:29:05.575 INFO:teuthology.orchestra.run.smithi124.stdout: Installing : python3-kubernetes-1:11.0.0-6.el8.noarch 12/13 2023-08-26T13:29:05.596 INFO:teuthology.orchestra.run.smithi031.stdout:(9/13): python3-oauthlib-2.1.0-1.el8.noarch.rpm 927 kB/s | 155 kB 00:00 2023-08-26T13:29:05.630 INFO:teuthology.orchestra.run.smithi031.stdout:(10/13): python3-pyasn1-modules-0.3.7-6.el8.noa 627 kB/s | 110 kB 00:00 2023-08-26T13:29:05.663 INFO:teuthology.orchestra.run.smithi031.stdout:(11/13): python3-jsonpatch-1.21-2.el8.noarch.rp 167 kB/s | 27 kB 00:00 2023-08-26T13:29:05.750 INFO:teuthology.orchestra.run.smithi124.stdout: Installing : ceph-mgr-rook-2:18.0.0-5706.ge7eb9853.el8.noarch 13/13 2023-08-26T13:29:05.789 INFO:teuthology.orchestra.run.smithi031.stdout:(12/13): python3-pyasn1-0.3.7-6.el8.noarch.rpm 658 kB/s | 126 kB 00:00 2023-08-26T13:29:05.805 INFO:teuthology.orchestra.run.smithi031.stdout:(13/13): python3-jsonpointer-1.10-11.el8.noarch 114 kB/s | 20 kB 00:00 2023-08-26T13:29:05.806 INFO:teuthology.orchestra.run.smithi031.stdout:-------------------------------------------------------------------------------- 2023-08-26T13:29:05.807 INFO:teuthology.orchestra.run.smithi031.stdout:Total 1.3 MB/s | 1.9 MB 00:01 2023-08-26T13:29:05.875 INFO:teuthology.orchestra.run.smithi031.stdout:Running transaction check 2023-08-26T13:29:05.893 INFO:teuthology.orchestra.run.smithi031.stdout:Transaction check succeeded. 2023-08-26T13:29:05.893 INFO:teuthology.orchestra.run.smithi031.stdout:Running transaction test 2023-08-26T13:29:05.971 INFO:teuthology.orchestra.run.smithi031.stdout:Transaction test succeeded. 2023-08-26T13:29:05.975 INFO:teuthology.orchestra.run.smithi031.stdout:Running transaction 2023-08-26T13:29:05.987 INFO:teuthology.orchestra.run.smithi124.stdout: Running scriptlet: ceph-mgr-rook-2:18.0.0-5706.ge7eb9853.el8.noarch 13/13 2023-08-26T13:29:05.987 INFO:teuthology.orchestra.run.smithi124.stdout: Verifying : ceph-mgr-rook-2:18.0.0-5706.ge7eb9853.el8.noarch 1/13 2023-08-26T13:29:05.987 INFO:teuthology.orchestra.run.smithi124.stdout: Verifying : python3-cachetools-3.1.1-4.el8.noarch 2/13 2023-08-26T13:29:05.987 INFO:teuthology.orchestra.run.smithi124.stdout: Verifying : python3-certifi-2018.10.15-7.el8.noarch 3/13 2023-08-26T13:29:05.987 INFO:teuthology.orchestra.run.smithi124.stdout: Verifying : python3-google-auth-1:1.1.1-10.el8.noarch 4/13 2023-08-26T13:29:05.988 INFO:teuthology.orchestra.run.smithi124.stdout: Verifying : python3-kubernetes-1:11.0.0-6.el8.noarch 5/13 2023-08-26T13:29:05.988 INFO:teuthology.orchestra.run.smithi124.stdout: Verifying : python3-rsa-4.9-2.el8.noarch 6/13 2023-08-26T13:29:05.988 INFO:teuthology.orchestra.run.smithi124.stdout: Verifying : python3-websocket-client-0.56.0-5.el8.noarch 7/13 2023-08-26T13:29:05.988 INFO:teuthology.orchestra.run.smithi124.stdout: Verifying : python3-requests-oauthlib-1.0.0-1.el8.noarch 8/13 2023-08-26T13:29:05.988 INFO:teuthology.orchestra.run.smithi124.stdout: Verifying : python3-oauthlib-2.1.0-1.el8.noarch 9/13 2023-08-26T13:29:05.989 INFO:teuthology.orchestra.run.smithi124.stdout: Verifying : python3-pyasn1-modules-0.3.7-6.el8.noarch 10/13 2023-08-26T13:29:05.989 INFO:teuthology.orchestra.run.smithi124.stdout: Verifying : python3-jsonpatch-1.21-2.el8.noarch 11/13 2023-08-26T13:29:05.989 INFO:teuthology.orchestra.run.smithi124.stdout: Verifying : python3-pyasn1-0.3.7-6.el8.noarch 12/13 2023-08-26T13:29:06.465 INFO:teuthology.orchestra.run.smithi031.stdout: Preparing : 1/1 2023-08-26T13:29:06.648 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : python3-pyasn1-0.3.7-6.el8.noarch 1/13 2023-08-26T13:29:06.859 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : python3-rsa-4.9-2.el8.noarch 2/13 2023-08-26T13:29:07.019 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : python3-pyasn1-modules-0.3.7-6.el8.noarch 3/13 2023-08-26T13:29:07.094 INFO:teuthology.orchestra.run.smithi124.stdout: Verifying : python3-jsonpointer-1.10-11.el8.noarch 13/13 2023-08-26T13:29:07.094 INFO:teuthology.orchestra.run.smithi124.stdout:Last metadata expiration check: 0:03:24 ago on Sat 26 Aug 2023 01:25:43 PM UTC. 2023-08-26T13:29:07.170 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : python3-jsonpointer-1.10-11.el8.noarch 4/13 2023-08-26T13:29:07.413 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : python3-jsonpatch-1.21-2.el8.noarch 5/13 2023-08-26T13:29:07.635 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : python3-oauthlib-2.1.0-1.el8.noarch 6/13 2023-08-26T13:29:07.842 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : python3-requests-oauthlib-1.0.0-1.el8.noarch 7/13 2023-08-26T13:29:08.028 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : python3-websocket-client-0.56.0-5.el8.noarch 8/13 2023-08-26T13:29:08.202 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : python3-certifi-2018.10.15-7.el8.noarch 9/13 2023-08-26T13:29:08.396 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : python3-cachetools-3.1.1-4.el8.noarch 10/13 2023-08-26T13:29:09.525 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : python3-google-auth-1:1.1.1-10.el8.noarch 11/13 2023-08-26T13:29:09.738 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : python3-kubernetes-1:11.0.0-6.el8.noarch 12/13 2023-08-26T13:29:09.862 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : ceph-mgr-rook-2:18.0.0-5706.ge7eb9853.el8.noarch 13/13 2023-08-26T13:29:10.110 INFO:teuthology.orchestra.run.smithi031.stdout: Running scriptlet: ceph-mgr-rook-2:18.0.0-5706.ge7eb9853.el8.noarch 13/13 2023-08-26T13:29:10.110 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : ceph-mgr-rook-2:18.0.0-5706.ge7eb9853.el8.noarch 1/13 2023-08-26T13:29:10.111 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : python3-cachetools-3.1.1-4.el8.noarch 2/13 2023-08-26T13:29:10.111 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : python3-certifi-2018.10.15-7.el8.noarch 3/13 2023-08-26T13:29:10.111 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : python3-google-auth-1:1.1.1-10.el8.noarch 4/13 2023-08-26T13:29:10.111 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : python3-kubernetes-1:11.0.0-6.el8.noarch 5/13 2023-08-26T13:29:10.111 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : python3-rsa-4.9-2.el8.noarch 6/13 2023-08-26T13:29:10.112 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : python3-websocket-client-0.56.0-5.el8.noarch 7/13 2023-08-26T13:29:10.112 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : python3-requests-oauthlib-1.0.0-1.el8.noarch 8/13 2023-08-26T13:29:10.112 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : python3-oauthlib-2.1.0-1.el8.noarch 9/13 2023-08-26T13:29:10.112 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : python3-pyasn1-modules-0.3.7-6.el8.noarch 10/13 2023-08-26T13:29:10.113 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : python3-jsonpatch-1.21-2.el8.noarch 11/13 2023-08-26T13:29:10.113 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : python3-pyasn1-0.3.7-6.el8.noarch 12/13 2023-08-26T13:29:11.251 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : python3-jsonpointer-1.10-11.el8.noarch 13/13 2023-08-26T13:29:11.252 INFO:teuthology.orchestra.run.smithi031.stdout:Last metadata expiration check: 0:03:28 ago on Sat 26 Aug 2023 01:25:43 PM UTC. 2023-08-26T13:29:11.848 INFO:teuthology.orchestra.run.smithi124.stdout:Installed products updated. 2023-08-26T13:29:12.033 INFO:teuthology.orchestra.run.smithi124.stdout: 2023-08-26T13:29:12.033 INFO:teuthology.orchestra.run.smithi124.stdout:Installed: 2023-08-26T13:29:12.034 INFO:teuthology.orchestra.run.smithi124.stdout: ceph-mgr-rook-2:18.0.0-5706.ge7eb9853.el8.noarch 2023-08-26T13:29:12.034 INFO:teuthology.orchestra.run.smithi124.stdout: python3-cachetools-3.1.1-4.el8.noarch 2023-08-26T13:29:12.034 INFO:teuthology.orchestra.run.smithi124.stdout: python3-certifi-2018.10.15-7.el8.noarch 2023-08-26T13:29:12.034 INFO:teuthology.orchestra.run.smithi124.stdout: python3-google-auth-1:1.1.1-10.el8.noarch 2023-08-26T13:29:12.034 INFO:teuthology.orchestra.run.smithi124.stdout: python3-jsonpatch-1.21-2.el8.noarch 2023-08-26T13:29:12.035 INFO:teuthology.orchestra.run.smithi124.stdout: python3-jsonpointer-1.10-11.el8.noarch 2023-08-26T13:29:12.035 INFO:teuthology.orchestra.run.smithi124.stdout: python3-kubernetes-1:11.0.0-6.el8.noarch 2023-08-26T13:29:12.035 INFO:teuthology.orchestra.run.smithi124.stdout: python3-oauthlib-2.1.0-1.el8.noarch 2023-08-26T13:29:12.035 INFO:teuthology.orchestra.run.smithi124.stdout: python3-pyasn1-0.3.7-6.el8.noarch 2023-08-26T13:29:12.035 INFO:teuthology.orchestra.run.smithi124.stdout: python3-pyasn1-modules-0.3.7-6.el8.noarch 2023-08-26T13:29:12.036 INFO:teuthology.orchestra.run.smithi124.stdout: python3-requests-oauthlib-1.0.0-1.el8.noarch 2023-08-26T13:29:12.036 INFO:teuthology.orchestra.run.smithi124.stdout: python3-rsa-4.9-2.el8.noarch 2023-08-26T13:29:12.036 INFO:teuthology.orchestra.run.smithi124.stdout: python3-websocket-client-0.56.0-5.el8.noarch 2023-08-26T13:29:12.036 INFO:teuthology.orchestra.run.smithi124.stdout: 2023-08-26T13:29:12.036 INFO:teuthology.orchestra.run.smithi124.stdout:Complete! 2023-08-26T13:29:12.176 DEBUG:teuthology.orchestra.run.smithi124:> sudo yum -y install ceph-mgr-cephadm 2023-08-26T13:29:12.529 INFO:teuthology.orchestra.run.smithi124.stdout:Updating Subscription Management repositories. 2023-08-26T13:29:14.146 INFO:teuthology.orchestra.run.smithi124.stdout:Last metadata expiration check: 0:03:31 ago on Sat 26 Aug 2023 01:25:43 PM UTC. 2023-08-26T13:29:15.533 INFO:teuthology.orchestra.run.smithi124.stdout:Dependencies resolved. 2023-08-26T13:29:15.534 INFO:teuthology.orchestra.run.smithi124.stdout:=============================================================================================== 2023-08-26T13:29:15.534 INFO:teuthology.orchestra.run.smithi124.stdout: Package Arch Version Repository Size 2023-08-26T13:29:15.534 INFO:teuthology.orchestra.run.smithi124.stdout:=============================================================================================== 2023-08-26T13:29:15.534 INFO:teuthology.orchestra.run.smithi124.stdout:Installing: 2023-08-26T13:29:15.535 INFO:teuthology.orchestra.run.smithi124.stdout: ceph-mgr-cephadm noarch 2:18.0.0-5706.ge7eb9853.el8 ceph-noarch 132 k 2023-08-26T13:29:15.535 INFO:teuthology.orchestra.run.smithi124.stdout:Installing dependencies: 2023-08-26T13:29:15.535 INFO:teuthology.orchestra.run.smithi124.stdout: python3-asyncssh noarch 2.7.0-2.el8 epel 476 k 2023-08-26T13:29:15.535 INFO:teuthology.orchestra.run.smithi124.stdout: python3-babel noarch 2.5.1-7.el8 rhel-8-for-x86_64-appstream-rpms 4.8 M 2023-08-26T13:29:15.535 INFO:teuthology.orchestra.run.smithi124.stdout: python3-jinja2 noarch 2.10.1-3.el8 rhel-8-for-x86_64-appstream-rpms 538 k 2023-08-26T13:29:15.535 INFO:teuthology.orchestra.run.smithi124.stdout: python3-natsort noarch 7.1.1-2.el8 epel 60 k 2023-08-26T13:29:15.536 INFO:teuthology.orchestra.run.smithi124.stdout: 2023-08-26T13:29:15.536 INFO:teuthology.orchestra.run.smithi124.stdout:Transaction Summary 2023-08-26T13:29:15.536 INFO:teuthology.orchestra.run.smithi124.stdout:=============================================================================================== 2023-08-26T13:29:15.536 INFO:teuthology.orchestra.run.smithi124.stdout:Install 5 Packages 2023-08-26T13:29:15.536 INFO:teuthology.orchestra.run.smithi124.stdout: 2023-08-26T13:29:15.537 INFO:teuthology.orchestra.run.smithi124.stdout:Total download size: 6.0 M 2023-08-26T13:29:15.537 INFO:teuthology.orchestra.run.smithi124.stdout:Installed size: 26 M 2023-08-26T13:29:15.538 INFO:teuthology.orchestra.run.smithi124.stdout:Downloading Packages: 2023-08-26T13:29:15.792 INFO:teuthology.orchestra.run.smithi124.stdout:(1/5): python3-natsort-7.1.1-2.el8.noarch.rpm 493 kB/s | 60 kB 00:00 2023-08-26T13:29:15.842 INFO:teuthology.orchestra.run.smithi124.stdout:(2/5): ceph-mgr-cephadm-18.0.0-5706.ge7eb9853.e 764 kB/s | 132 kB 00:00 2023-08-26T13:29:15.867 INFO:teuthology.orchestra.run.smithi124.stdout:(3/5): python3-asyncssh-2.7.0-2.el8.noarch.rpm 2.3 MB/s | 476 kB 00:00 2023-08-26T13:29:15.901 INFO:teuthology.orchestra.run.smithi124.stdout:(4/5): python3-jinja2-2.10.1-3.el8.noarch.rpm 4.8 MB/s | 538 kB 00:00 2023-08-26T13:29:15.993 INFO:teuthology.orchestra.run.smithi124.stdout:(5/5): python3-babel-2.5.1-7.el8.noarch.rpm 32 MB/s | 4.8 MB 00:00 2023-08-26T13:29:15.994 INFO:teuthology.orchestra.run.smithi124.stdout:-------------------------------------------------------------------------------- 2023-08-26T13:29:15.994 INFO:teuthology.orchestra.run.smithi124.stdout:Total 13 MB/s | 6.0 MB 00:00 2023-08-26T13:29:16.047 INFO:teuthology.orchestra.run.smithi124.stdout:Running transaction check 2023-08-26T13:29:16.058 INFO:teuthology.orchestra.run.smithi124.stdout:Transaction check succeeded. 2023-08-26T13:29:16.058 INFO:teuthology.orchestra.run.smithi124.stdout:Running transaction test 2023-08-26T13:29:16.138 INFO:teuthology.orchestra.run.smithi124.stdout:Transaction test succeeded. 2023-08-26T13:29:16.142 INFO:teuthology.orchestra.run.smithi124.stdout:Running transaction 2023-08-26T13:29:16.352 INFO:teuthology.orchestra.run.smithi031.stdout:Installed products updated. 2023-08-26T13:29:16.554 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-08-26T13:29:16.554 INFO:teuthology.orchestra.run.smithi031.stdout:Installed: 2023-08-26T13:29:16.554 INFO:teuthology.orchestra.run.smithi031.stdout: ceph-mgr-rook-2:18.0.0-5706.ge7eb9853.el8.noarch 2023-08-26T13:29:16.554 INFO:teuthology.orchestra.run.smithi031.stdout: python3-cachetools-3.1.1-4.el8.noarch 2023-08-26T13:29:16.555 INFO:teuthology.orchestra.run.smithi031.stdout: python3-certifi-2018.10.15-7.el8.noarch 2023-08-26T13:29:16.555 INFO:teuthology.orchestra.run.smithi031.stdout: python3-google-auth-1:1.1.1-10.el8.noarch 2023-08-26T13:29:16.555 INFO:teuthology.orchestra.run.smithi031.stdout: python3-jsonpatch-1.21-2.el8.noarch 2023-08-26T13:29:16.555 INFO:teuthology.orchestra.run.smithi031.stdout: python3-jsonpointer-1.10-11.el8.noarch 2023-08-26T13:29:16.555 INFO:teuthology.orchestra.run.smithi031.stdout: python3-kubernetes-1:11.0.0-6.el8.noarch 2023-08-26T13:29:16.556 INFO:teuthology.orchestra.run.smithi031.stdout: python3-oauthlib-2.1.0-1.el8.noarch 2023-08-26T13:29:16.556 INFO:teuthology.orchestra.run.smithi031.stdout: python3-pyasn1-0.3.7-6.el8.noarch 2023-08-26T13:29:16.556 INFO:teuthology.orchestra.run.smithi031.stdout: python3-pyasn1-modules-0.3.7-6.el8.noarch 2023-08-26T13:29:16.556 INFO:teuthology.orchestra.run.smithi031.stdout: python3-requests-oauthlib-1.0.0-1.el8.noarch 2023-08-26T13:29:16.556 INFO:teuthology.orchestra.run.smithi031.stdout: python3-rsa-4.9-2.el8.noarch 2023-08-26T13:29:16.557 INFO:teuthology.orchestra.run.smithi031.stdout: python3-websocket-client-0.56.0-5.el8.noarch 2023-08-26T13:29:16.557 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-08-26T13:29:16.557 INFO:teuthology.orchestra.run.smithi031.stdout:Complete! 2023-08-26T13:29:16.746 DEBUG:teuthology.orchestra.run.smithi031:> sudo yum -y install ceph-mgr-cephadm 2023-08-26T13:29:17.119 INFO:teuthology.orchestra.run.smithi031.stdout:Updating Subscription Management repositories. 2023-08-26T13:29:17.871 INFO:teuthology.orchestra.run.smithi124.stdout: Preparing : 1/1 2023-08-26T13:29:18.233 INFO:teuthology.orchestra.run.smithi124.stdout: Installing : python3-babel-2.5.1-7.el8.noarch 1/5 2023-08-26T13:29:18.441 INFO:teuthology.orchestra.run.smithi124.stdout: Installing : python3-jinja2-2.10.1-3.el8.noarch 2/5 2023-08-26T13:29:18.710 INFO:teuthology.orchestra.run.smithi031.stdout:Last metadata expiration check: 0:03:35 ago on Sat 26 Aug 2023 01:25:43 PM UTC. 2023-08-26T13:29:18.793 INFO:teuthology.orchestra.run.smithi124.stdout: Installing : python3-natsort-7.1.1-2.el8.noarch 3/5 2023-08-26T13:29:19.079 INFO:teuthology.orchestra.run.smithi124.stdout: Installing : python3-asyncssh-2.7.0-2.el8.noarch 4/5 2023-08-26T13:29:19.306 INFO:teuthology.orchestra.run.smithi124.stdout: Installing : ceph-mgr-cephadm-2:18.0.0-5706.ge7eb9853.el8.noarch 5/5 2023-08-26T13:29:19.543 INFO:teuthology.orchestra.run.smithi124.stdout: Running scriptlet: ceph-mgr-cephadm-2:18.0.0-5706.ge7eb9853.el8.noarch 5/5 2023-08-26T13:29:19.543 INFO:teuthology.orchestra.run.smithi124.stdout: Verifying : ceph-mgr-cephadm-2:18.0.0-5706.ge7eb9853.el8.noarch 1/5 2023-08-26T13:29:19.544 INFO:teuthology.orchestra.run.smithi124.stdout: Verifying : python3-asyncssh-2.7.0-2.el8.noarch 2/5 2023-08-26T13:29:19.544 INFO:teuthology.orchestra.run.smithi124.stdout: Verifying : python3-natsort-7.1.1-2.el8.noarch 3/5 2023-08-26T13:29:19.544 INFO:teuthology.orchestra.run.smithi124.stdout: Verifying : python3-jinja2-2.10.1-3.el8.noarch 4/5 2023-08-26T13:29:20.249 INFO:teuthology.orchestra.run.smithi031.stdout:Dependencies resolved. 2023-08-26T13:29:20.250 INFO:teuthology.orchestra.run.smithi031.stdout:=============================================================================================== 2023-08-26T13:29:20.250 INFO:teuthology.orchestra.run.smithi031.stdout: Package Arch Version Repository Size 2023-08-26T13:29:20.250 INFO:teuthology.orchestra.run.smithi031.stdout:=============================================================================================== 2023-08-26T13:29:20.251 INFO:teuthology.orchestra.run.smithi031.stdout:Installing: 2023-08-26T13:29:20.251 INFO:teuthology.orchestra.run.smithi031.stdout: ceph-mgr-cephadm noarch 2:18.0.0-5706.ge7eb9853.el8 ceph-noarch 132 k 2023-08-26T13:29:20.251 INFO:teuthology.orchestra.run.smithi031.stdout:Installing dependencies: 2023-08-26T13:29:20.251 INFO:teuthology.orchestra.run.smithi031.stdout: python3-asyncssh noarch 2.7.0-2.el8 epel 476 k 2023-08-26T13:29:20.251 INFO:teuthology.orchestra.run.smithi031.stdout: python3-babel noarch 2.5.1-7.el8 rhel-8-for-x86_64-appstream-rpms 4.8 M 2023-08-26T13:29:20.252 INFO:teuthology.orchestra.run.smithi031.stdout: python3-jinja2 noarch 2.10.1-3.el8 rhel-8-for-x86_64-appstream-rpms 538 k 2023-08-26T13:29:20.252 INFO:teuthology.orchestra.run.smithi031.stdout: python3-natsort noarch 7.1.1-2.el8 epel 60 k 2023-08-26T13:29:20.252 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-08-26T13:29:20.252 INFO:teuthology.orchestra.run.smithi031.stdout:Transaction Summary 2023-08-26T13:29:20.252 INFO:teuthology.orchestra.run.smithi031.stdout:=============================================================================================== 2023-08-26T13:29:20.253 INFO:teuthology.orchestra.run.smithi031.stdout:Install 5 Packages 2023-08-26T13:29:20.253 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-08-26T13:29:20.253 INFO:teuthology.orchestra.run.smithi031.stdout:Total download size: 6.0 M 2023-08-26T13:29:20.254 INFO:teuthology.orchestra.run.smithi031.stdout:Installed size: 26 M 2023-08-26T13:29:20.254 INFO:teuthology.orchestra.run.smithi031.stdout:Downloading Packages: 2023-08-26T13:29:20.520 INFO:teuthology.orchestra.run.smithi031.stdout:(1/5): ceph-mgr-cephadm-18.0.0-5706.ge7eb9853.e 717 kB/s | 132 kB 00:00 2023-08-26T13:29:20.537 INFO:teuthology.orchestra.run.smithi031.stdout:(2/5): python3-natsort-7.1.1-2.el8.noarch.rpm 300 kB/s | 60 kB 00:00 2023-08-26T13:29:20.639 INFO:teuthology.orchestra.run.smithi124.stdout: Verifying : python3-babel-2.5.1-7.el8.noarch 5/5 2023-08-26T13:29:20.639 INFO:teuthology.orchestra.run.smithi124.stdout:Last metadata expiration check: 0:03:37 ago on Sat 26 Aug 2023 01:25:43 PM UTC. 2023-08-26T13:29:20.671 INFO:teuthology.orchestra.run.smithi031.stdout:(3/5): python3-asyncssh-2.7.0-2.el8.noarch.rpm 1.4 MB/s | 476 kB 00:00 2023-08-26T13:29:20.749 INFO:teuthology.orchestra.run.smithi031.stdout:(4/5): python3-jinja2-2.10.1-3.el8.noarch.rpm 2.3 MB/s | 538 kB 00:00 2023-08-26T13:29:20.856 INFO:teuthology.orchestra.run.smithi031.stdout:(5/5): python3-babel-2.5.1-7.el8.noarch.rpm 15 MB/s | 4.8 MB 00:00 2023-08-26T13:29:20.857 INFO:teuthology.orchestra.run.smithi031.stdout:-------------------------------------------------------------------------------- 2023-08-26T13:29:20.857 INFO:teuthology.orchestra.run.smithi031.stdout:Total 9.8 MB/s | 6.0 MB 00:00 2023-08-26T13:29:20.912 INFO:teuthology.orchestra.run.smithi031.stdout:Running transaction check 2023-08-26T13:29:20.924 INFO:teuthology.orchestra.run.smithi031.stdout:Transaction check succeeded. 2023-08-26T13:29:20.924 INFO:teuthology.orchestra.run.smithi031.stdout:Running transaction test 2023-08-26T13:29:21.009 INFO:teuthology.orchestra.run.smithi031.stdout:Transaction test succeeded. 2023-08-26T13:29:21.013 INFO:teuthology.orchestra.run.smithi031.stdout:Running transaction 2023-08-26T13:29:22.042 INFO:teuthology.orchestra.run.smithi031.stdout: Preparing : 1/1 2023-08-26T13:29:22.814 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : python3-babel-2.5.1-7.el8.noarch 1/5 2023-08-26T13:29:23.013 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : python3-jinja2-2.10.1-3.el8.noarch 2/5 2023-08-26T13:29:23.316 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : python3-natsort-7.1.1-2.el8.noarch 3/5 2023-08-26T13:29:23.586 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : python3-asyncssh-2.7.0-2.el8.noarch 4/5 2023-08-26T13:29:23.793 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : ceph-mgr-cephadm-2:18.0.0-5706.ge7eb9853.el8.noarch 5/5 2023-08-26T13:29:24.052 INFO:teuthology.orchestra.run.smithi031.stdout: Running scriptlet: ceph-mgr-cephadm-2:18.0.0-5706.ge7eb9853.el8.noarch 5/5 2023-08-26T13:29:24.053 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : ceph-mgr-cephadm-2:18.0.0-5706.ge7eb9853.el8.noarch 1/5 2023-08-26T13:29:24.053 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : python3-asyncssh-2.7.0-2.el8.noarch 2/5 2023-08-26T13:29:24.053 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : python3-natsort-7.1.1-2.el8.noarch 3/5 2023-08-26T13:29:24.053 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : python3-jinja2-2.10.1-3.el8.noarch 4/5 2023-08-26T13:29:25.133 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : python3-babel-2.5.1-7.el8.noarch 5/5 2023-08-26T13:29:25.134 INFO:teuthology.orchestra.run.smithi031.stdout:Last metadata expiration check: 0:03:42 ago on Sat 26 Aug 2023 01:25:43 PM UTC. 2023-08-26T13:29:27.243 INFO:teuthology.orchestra.run.smithi124.stdout:Installed products updated. 2023-08-26T13:29:27.427 INFO:teuthology.orchestra.run.smithi124.stdout: 2023-08-26T13:29:27.427 INFO:teuthology.orchestra.run.smithi124.stdout:Installed: 2023-08-26T13:29:27.427 INFO:teuthology.orchestra.run.smithi124.stdout: ceph-mgr-cephadm-2:18.0.0-5706.ge7eb9853.el8.noarch 2023-08-26T13:29:27.427 INFO:teuthology.orchestra.run.smithi124.stdout: python3-asyncssh-2.7.0-2.el8.noarch 2023-08-26T13:29:27.428 INFO:teuthology.orchestra.run.smithi124.stdout: python3-babel-2.5.1-7.el8.noarch 2023-08-26T13:29:27.428 INFO:teuthology.orchestra.run.smithi124.stdout: python3-jinja2-2.10.1-3.el8.noarch 2023-08-26T13:29:27.428 INFO:teuthology.orchestra.run.smithi124.stdout: python3-natsort-7.1.1-2.el8.noarch 2023-08-26T13:29:27.428 INFO:teuthology.orchestra.run.smithi124.stdout: 2023-08-26T13:29:27.428 INFO:teuthology.orchestra.run.smithi124.stdout:Complete! 2023-08-26T13:29:27.538 DEBUG:teuthology.orchestra.run.smithi124:> sudo yum -y install ceph-fuse 2023-08-26T13:29:27.889 INFO:teuthology.orchestra.run.smithi124.stdout:Updating Subscription Management repositories. 2023-08-26T13:29:29.056 INFO:teuthology.orchestra.run.smithi124.stdout:Last metadata expiration check: 0:03:46 ago on Sat 26 Aug 2023 01:25:43 PM UTC. 2023-08-26T13:29:29.104 INFO:teuthology.orchestra.run.smithi031.stdout:Installed products updated. 2023-08-26T13:29:29.305 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-08-26T13:29:29.305 INFO:teuthology.orchestra.run.smithi031.stdout:Installed: 2023-08-26T13:29:29.306 INFO:teuthology.orchestra.run.smithi031.stdout: ceph-mgr-cephadm-2:18.0.0-5706.ge7eb9853.el8.noarch 2023-08-26T13:29:29.306 INFO:teuthology.orchestra.run.smithi031.stdout: python3-asyncssh-2.7.0-2.el8.noarch 2023-08-26T13:29:29.306 INFO:teuthology.orchestra.run.smithi031.stdout: python3-babel-2.5.1-7.el8.noarch 2023-08-26T13:29:29.307 INFO:teuthology.orchestra.run.smithi031.stdout: python3-jinja2-2.10.1-3.el8.noarch 2023-08-26T13:29:29.307 INFO:teuthology.orchestra.run.smithi031.stdout: python3-natsort-7.1.1-2.el8.noarch 2023-08-26T13:29:29.307 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-08-26T13:29:29.308 INFO:teuthology.orchestra.run.smithi031.stdout:Complete! 2023-08-26T13:29:29.444 DEBUG:teuthology.orchestra.run.smithi031:> sudo yum -y install ceph-fuse 2023-08-26T13:29:29.819 INFO:teuthology.orchestra.run.smithi031.stdout:Updating Subscription Management repositories. 2023-08-26T13:29:30.440 INFO:teuthology.orchestra.run.smithi124.stdout:Dependencies resolved. 2023-08-26T13:29:30.440 INFO:teuthology.orchestra.run.smithi124.stdout:================================================================================ 2023-08-26T13:29:30.440 INFO:teuthology.orchestra.run.smithi124.stdout: Package Arch Version Repository Size 2023-08-26T13:29:30.441 INFO:teuthology.orchestra.run.smithi124.stdout:================================================================================ 2023-08-26T13:29:30.441 INFO:teuthology.orchestra.run.smithi124.stdout:Installing: 2023-08-26T13:29:30.441 INFO:teuthology.orchestra.run.smithi124.stdout: ceph-fuse 2023-08-26T13:29:30.441 INFO:teuthology.orchestra.run.smithi124.stdout: x86_64 2:18.0.0-5706.ge7eb9853.el8 ceph 862 k 2023-08-26T13:29:30.442 INFO:teuthology.orchestra.run.smithi124.stdout:Installing dependencies: 2023-08-26T13:29:30.442 INFO:teuthology.orchestra.run.smithi124.stdout: fuse x86_64 2.9.7-15.el8 rhel-8-for-x86_64-baseos-rpms 83 k 2023-08-26T13:29:30.442 INFO:teuthology.orchestra.run.smithi124.stdout: 2023-08-26T13:29:30.442 INFO:teuthology.orchestra.run.smithi124.stdout:Transaction Summary 2023-08-26T13:29:30.442 INFO:teuthology.orchestra.run.smithi124.stdout:================================================================================ 2023-08-26T13:29:30.442 INFO:teuthology.orchestra.run.smithi124.stdout:Install 2 Packages 2023-08-26T13:29:30.443 INFO:teuthology.orchestra.run.smithi124.stdout: 2023-08-26T13:29:30.443 INFO:teuthology.orchestra.run.smithi124.stdout:Total download size: 944 k 2023-08-26T13:29:30.443 INFO:teuthology.orchestra.run.smithi124.stdout:Installed size: 3.0 M 2023-08-26T13:29:30.444 INFO:teuthology.orchestra.run.smithi124.stdout:Downloading Packages: 2023-08-26T13:29:30.515 INFO:teuthology.orchestra.run.smithi124.stdout:(1/2): fuse-2.9.7-15.el8.x86_64.rpm 1.1 MB/s | 83 kB 00:00 2023-08-26T13:29:30.820 INFO:teuthology.orchestra.run.smithi031.stdout:Last metadata expiration check: 0:03:47 ago on Sat 26 Aug 2023 01:25:43 PM UTC. 2023-08-26T13:29:30.874 INFO:teuthology.orchestra.run.smithi124.stdout:(2/2): ceph-fuse-18.0.0-5706.ge7eb9853.el8.x86_ 1.9 MB/s | 862 kB 00:00 2023-08-26T13:29:30.874 INFO:teuthology.orchestra.run.smithi124.stdout:-------------------------------------------------------------------------------- 2023-08-26T13:29:30.875 INFO:teuthology.orchestra.run.smithi124.stdout:Total 2.1 MB/s | 944 kB 00:00 2023-08-26T13:29:30.886 INFO:teuthology.orchestra.run.smithi124.stdout:Running transaction check 2023-08-26T13:29:30.907 INFO:teuthology.orchestra.run.smithi124.stdout:Transaction check succeeded. 2023-08-26T13:29:30.908 INFO:teuthology.orchestra.run.smithi124.stdout:Running transaction test 2023-08-26T13:29:30.980 INFO:teuthology.orchestra.run.smithi124.stdout:Transaction test succeeded. 2023-08-26T13:29:30.984 INFO:teuthology.orchestra.run.smithi124.stdout:Running transaction 2023-08-26T13:29:31.162 INFO:teuthology.orchestra.run.smithi124.stdout: Preparing : 1/1 2023-08-26T13:29:31.412 INFO:teuthology.orchestra.run.smithi124.stdout: Installing : fuse-2.9.7-15.el8.x86_64 1/2 2023-08-26T13:29:31.586 INFO:teuthology.orchestra.run.smithi124.stdout: Installing : ceph-fuse-2:18.0.0-5706.ge7eb9853.el8.x86_64 2/2 2023-08-26T13:29:31.939 INFO:teuthology.orchestra.run.smithi124.stdout: Running scriptlet: ceph-fuse-2:18.0.0-5706.ge7eb9853.el8.x86_64 2/2 2023-08-26T13:29:31.939 INFO:teuthology.orchestra.run.smithi124.stdout: Verifying : ceph-fuse-2:18.0.0-5706.ge7eb9853.el8.x86_64 1/2 2023-08-26T13:29:32.420 INFO:teuthology.orchestra.run.smithi031.stdout:Dependencies resolved. 2023-08-26T13:29:32.421 INFO:teuthology.orchestra.run.smithi031.stdout:================================================================================ 2023-08-26T13:29:32.421 INFO:teuthology.orchestra.run.smithi031.stdout: Package Arch Version Repository Size 2023-08-26T13:29:32.421 INFO:teuthology.orchestra.run.smithi031.stdout:================================================================================ 2023-08-26T13:29:32.421 INFO:teuthology.orchestra.run.smithi031.stdout:Installing: 2023-08-26T13:29:32.422 INFO:teuthology.orchestra.run.smithi031.stdout: ceph-fuse 2023-08-26T13:29:32.422 INFO:teuthology.orchestra.run.smithi031.stdout: x86_64 2:18.0.0-5706.ge7eb9853.el8 ceph 862 k 2023-08-26T13:29:32.422 INFO:teuthology.orchestra.run.smithi031.stdout:Installing dependencies: 2023-08-26T13:29:32.422 INFO:teuthology.orchestra.run.smithi031.stdout: fuse x86_64 2.9.7-15.el8 rhel-8-for-x86_64-baseos-rpms 83 k 2023-08-26T13:29:32.422 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-08-26T13:29:32.423 INFO:teuthology.orchestra.run.smithi031.stdout:Transaction Summary 2023-08-26T13:29:32.423 INFO:teuthology.orchestra.run.smithi031.stdout:================================================================================ 2023-08-26T13:29:32.423 INFO:teuthology.orchestra.run.smithi031.stdout:Install 2 Packages 2023-08-26T13:29:32.423 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-08-26T13:29:32.424 INFO:teuthology.orchestra.run.smithi031.stdout:Total download size: 944 k 2023-08-26T13:29:32.424 INFO:teuthology.orchestra.run.smithi031.stdout:Installed size: 3.0 M 2023-08-26T13:29:32.424 INFO:teuthology.orchestra.run.smithi031.stdout:Downloading Packages: 2023-08-26T13:29:32.500 INFO:teuthology.orchestra.run.smithi031.stdout:(1/2): fuse-2.9.7-15.el8.x86_64.rpm 1.0 MB/s | 83 kB 00:00 2023-08-26T13:29:32.999 INFO:teuthology.orchestra.run.smithi124.stdout: Verifying : fuse-2.9.7-15.el8.x86_64 2/2 2023-08-26T13:29:32.999 INFO:teuthology.orchestra.run.smithi124.stdout:Last metadata expiration check: 0:03:49 ago on Sat 26 Aug 2023 01:25:43 PM UTC. 2023-08-26T13:29:33.000 INFO:teuthology.orchestra.run.smithi031.stdout:(2/2): ceph-fuse-18.0.0-5706.ge7eb9853.el8.x86_ 1.5 MB/s | 862 kB 00:00 2023-08-26T13:29:33.001 INFO:teuthology.orchestra.run.smithi031.stdout:-------------------------------------------------------------------------------- 2023-08-26T13:29:33.002 INFO:teuthology.orchestra.run.smithi031.stdout:Total 1.6 MB/s | 944 kB 00:00 2023-08-26T13:29:33.014 INFO:teuthology.orchestra.run.smithi031.stdout:Running transaction check 2023-08-26T13:29:33.037 INFO:teuthology.orchestra.run.smithi031.stdout:Transaction check succeeded. 2023-08-26T13:29:33.038 INFO:teuthology.orchestra.run.smithi031.stdout:Running transaction test 2023-08-26T13:29:33.117 INFO:teuthology.orchestra.run.smithi031.stdout:Transaction test succeeded. 2023-08-26T13:29:33.121 INFO:teuthology.orchestra.run.smithi031.stdout:Running transaction 2023-08-26T13:29:33.322 INFO:teuthology.orchestra.run.smithi031.stdout: Preparing : 1/1 2023-08-26T13:29:33.554 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : fuse-2.9.7-15.el8.x86_64 1/2 2023-08-26T13:29:33.707 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : ceph-fuse-2:18.0.0-5706.ge7eb9853.el8.x86_64 2/2 2023-08-26T13:29:34.056 INFO:teuthology.orchestra.run.smithi031.stdout: Running scriptlet: ceph-fuse-2:18.0.0-5706.ge7eb9853.el8.x86_64 2/2 2023-08-26T13:29:34.056 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : ceph-fuse-2:18.0.0-5706.ge7eb9853.el8.x86_64 1/2 2023-08-26T13:29:35.148 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : fuse-2.9.7-15.el8.x86_64 2/2 2023-08-26T13:29:35.148 INFO:teuthology.orchestra.run.smithi031.stdout:Last metadata expiration check: 0:03:52 ago on Sat 26 Aug 2023 01:25:43 PM UTC. 2023-08-26T13:29:35.397 INFO:teuthology.orchestra.run.smithi124.stdout:Installed products updated. 2023-08-26T13:29:35.581 INFO:teuthology.orchestra.run.smithi124.stdout: 2023-08-26T13:29:35.582 INFO:teuthology.orchestra.run.smithi124.stdout:Installed: 2023-08-26T13:29:35.582 INFO:teuthology.orchestra.run.smithi124.stdout: ceph-fuse-2:18.0.0-5706.ge7eb9853.el8.x86_64 fuse-2.9.7-15.el8.x86_64 2023-08-26T13:29:35.582 INFO:teuthology.orchestra.run.smithi124.stdout: 2023-08-26T13:29:35.582 INFO:teuthology.orchestra.run.smithi124.stdout:Complete! 2023-08-26T13:29:35.704 DEBUG:teuthology.orchestra.run.smithi124:> sudo yum -y install ceph-volume 2023-08-26T13:29:36.060 INFO:teuthology.orchestra.run.smithi124.stdout:Updating Subscription Management repositories. 2023-08-26T13:29:36.906 INFO:teuthology.orchestra.run.smithi124.stdout:Last metadata expiration check: 0:03:53 ago on Sat 26 Aug 2023 01:25:43 PM UTC. 2023-08-26T13:29:37.953 INFO:teuthology.orchestra.run.smithi031.stdout:Installed products updated. 2023-08-26T13:29:38.153 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-08-26T13:29:38.153 INFO:teuthology.orchestra.run.smithi031.stdout:Installed: 2023-08-26T13:29:38.154 INFO:teuthology.orchestra.run.smithi031.stdout: ceph-fuse-2:18.0.0-5706.ge7eb9853.el8.x86_64 fuse-2.9.7-15.el8.x86_64 2023-08-26T13:29:38.154 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-08-26T13:29:38.154 INFO:teuthology.orchestra.run.smithi031.stdout:Complete! 2023-08-26T13:29:38.292 DEBUG:teuthology.orchestra.run.smithi031:> sudo yum -y install ceph-volume 2023-08-26T13:29:38.323 INFO:teuthology.orchestra.run.smithi124.stdout:Dependencies resolved. 2023-08-26T13:29:38.323 INFO:teuthology.orchestra.run.smithi124.stdout:================================================================================ 2023-08-26T13:29:38.324 INFO:teuthology.orchestra.run.smithi124.stdout: Package Arch Version Repository Size 2023-08-26T13:29:38.324 INFO:teuthology.orchestra.run.smithi124.stdout:================================================================================ 2023-08-26T13:29:38.324 INFO:teuthology.orchestra.run.smithi124.stdout:Installing: 2023-08-26T13:29:38.324 INFO:teuthology.orchestra.run.smithi124.stdout: ceph-volume noarch 2:18.0.0-5706.ge7eb9853.el8 ceph-noarch 260 k 2023-08-26T13:29:38.324 INFO:teuthology.orchestra.run.smithi124.stdout: 2023-08-26T13:29:38.325 INFO:teuthology.orchestra.run.smithi124.stdout:Transaction Summary 2023-08-26T13:29:38.325 INFO:teuthology.orchestra.run.smithi124.stdout:================================================================================ 2023-08-26T13:29:38.325 INFO:teuthology.orchestra.run.smithi124.stdout:Install 1 Package 2023-08-26T13:29:38.325 INFO:teuthology.orchestra.run.smithi124.stdout: 2023-08-26T13:29:38.326 INFO:teuthology.orchestra.run.smithi124.stdout:Total download size: 260 k 2023-08-26T13:29:38.326 INFO:teuthology.orchestra.run.smithi124.stdout:Installed size: 1.3 M 2023-08-26T13:29:38.326 INFO:teuthology.orchestra.run.smithi124.stdout:Downloading Packages: 2023-08-26T13:29:38.532 INFO:teuthology.orchestra.run.smithi124.stdout:ceph-volume-18.0.0-5706.ge7eb9853.el8.noarch.rp 1.2 MB/s | 260 kB 00:00 2023-08-26T13:29:38.569 INFO:teuthology.orchestra.run.smithi124.stdout:-------------------------------------------------------------------------------- 2023-08-26T13:29:38.570 INFO:teuthology.orchestra.run.smithi124.stdout:Total 1.2 MB/s | 260 kB 00:00 2023-08-26T13:29:38.570 INFO:teuthology.orchestra.run.smithi124.stdout:Running transaction check 2023-08-26T13:29:38.570 INFO:teuthology.orchestra.run.smithi124.stdout:Transaction check succeeded. 2023-08-26T13:29:38.571 INFO:teuthology.orchestra.run.smithi124.stdout:Running transaction test 2023-08-26T13:29:38.571 INFO:teuthology.orchestra.run.smithi124.stdout:Transaction test succeeded. 2023-08-26T13:29:38.571 INFO:teuthology.orchestra.run.smithi124.stdout:Running transaction 2023-08-26T13:29:38.670 INFO:teuthology.orchestra.run.smithi031.stdout:Updating Subscription Management repositories. 2023-08-26T13:29:38.731 INFO:teuthology.orchestra.run.smithi124.stdout: Preparing : 1/1 2023-08-26T13:29:39.329 INFO:teuthology.orchestra.run.smithi124.stdout: Installing : ceph-volume-2:18.0.0-5706.ge7eb9853.el8.noarch 1/1 2023-08-26T13:29:39.583 INFO:teuthology.orchestra.run.smithi031.stdout:Last metadata expiration check: 0:03:56 ago on Sat 26 Aug 2023 01:25:43 PM UTC. 2023-08-26T13:29:39.678 INFO:teuthology.orchestra.run.smithi124.stdout: Running scriptlet: ceph-volume-2:18.0.0-5706.ge7eb9853.el8.noarch 1/1 2023-08-26T13:29:40.704 INFO:teuthology.orchestra.run.smithi124.stdout: Verifying : ceph-volume-2:18.0.0-5706.ge7eb9853.el8.noarch 1/1 2023-08-26T13:29:40.704 INFO:teuthology.orchestra.run.smithi124.stdout:Last metadata expiration check: 0:03:57 ago on Sat 26 Aug 2023 01:25:43 PM UTC. 2023-08-26T13:29:41.133 INFO:teuthology.orchestra.run.smithi031.stdout:Dependencies resolved. 2023-08-26T13:29:41.134 INFO:teuthology.orchestra.run.smithi031.stdout:================================================================================ 2023-08-26T13:29:41.134 INFO:teuthology.orchestra.run.smithi031.stdout: Package Arch Version Repository Size 2023-08-26T13:29:41.135 INFO:teuthology.orchestra.run.smithi031.stdout:================================================================================ 2023-08-26T13:29:41.135 INFO:teuthology.orchestra.run.smithi031.stdout:Installing: 2023-08-26T13:29:41.135 INFO:teuthology.orchestra.run.smithi031.stdout: ceph-volume noarch 2:18.0.0-5706.ge7eb9853.el8 ceph-noarch 260 k 2023-08-26T13:29:41.135 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-08-26T13:29:41.135 INFO:teuthology.orchestra.run.smithi031.stdout:Transaction Summary 2023-08-26T13:29:41.136 INFO:teuthology.orchestra.run.smithi031.stdout:================================================================================ 2023-08-26T13:29:41.136 INFO:teuthology.orchestra.run.smithi031.stdout:Install 1 Package 2023-08-26T13:29:41.136 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-08-26T13:29:41.136 INFO:teuthology.orchestra.run.smithi031.stdout:Total download size: 260 k 2023-08-26T13:29:41.136 INFO:teuthology.orchestra.run.smithi031.stdout:Installed size: 1.3 M 2023-08-26T13:29:41.137 INFO:teuthology.orchestra.run.smithi031.stdout:Downloading Packages: 2023-08-26T13:29:41.580 INFO:teuthology.orchestra.run.smithi031.stdout:ceph-volume-18.0.0-5706.ge7eb9853.el8.noarch.rp 586 kB/s | 260 kB 00:00 2023-08-26T13:29:41.580 INFO:teuthology.orchestra.run.smithi031.stdout:-------------------------------------------------------------------------------- 2023-08-26T13:29:41.581 INFO:teuthology.orchestra.run.smithi031.stdout:Total 584 kB/s | 260 kB 00:00 2023-08-26T13:29:41.581 INFO:teuthology.orchestra.run.smithi031.stdout:Running transaction check 2023-08-26T13:29:41.592 INFO:teuthology.orchestra.run.smithi031.stdout:Transaction check succeeded. 2023-08-26T13:29:41.593 INFO:teuthology.orchestra.run.smithi031.stdout:Running transaction test 2023-08-26T13:29:41.612 INFO:teuthology.orchestra.run.smithi031.stdout:Transaction test succeeded. 2023-08-26T13:29:41.616 INFO:teuthology.orchestra.run.smithi031.stdout:Running transaction 2023-08-26T13:29:41.793 INFO:teuthology.orchestra.run.smithi031.stdout: Preparing : 1/1 2023-08-26T13:29:42.015 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : ceph-volume-2:18.0.0-5706.ge7eb9853.el8.noarch 1/1 2023-08-26T13:29:42.389 INFO:teuthology.orchestra.run.smithi031.stdout: Running scriptlet: ceph-volume-2:18.0.0-5706.ge7eb9853.el8.noarch 1/1 2023-08-26T13:29:43.692 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : ceph-volume-2:18.0.0-5706.ge7eb9853.el8.noarch 1/1 2023-08-26T13:29:43.693 INFO:teuthology.orchestra.run.smithi031.stdout:Last metadata expiration check: 0:04:00 ago on Sat 26 Aug 2023 01:25:43 PM UTC. 2023-08-26T13:29:43.700 INFO:teuthology.orchestra.run.smithi124.stdout:Installed products updated. 2023-08-26T13:29:43.884 INFO:teuthology.orchestra.run.smithi124.stdout: 2023-08-26T13:29:43.884 INFO:teuthology.orchestra.run.smithi124.stdout:Installed: 2023-08-26T13:29:43.884 INFO:teuthology.orchestra.run.smithi124.stdout: ceph-volume-2:18.0.0-5706.ge7eb9853.el8.noarch 2023-08-26T13:29:43.884 INFO:teuthology.orchestra.run.smithi124.stdout: 2023-08-26T13:29:43.885 INFO:teuthology.orchestra.run.smithi124.stdout:Complete! 2023-08-26T13:29:44.018 DEBUG:teuthology.orchestra.run.smithi124:> sudo yum -y install librados-devel 2023-08-26T13:29:44.374 INFO:teuthology.orchestra.run.smithi124.stdout:Updating Subscription Management repositories. 2023-08-26T13:29:45.193 INFO:teuthology.orchestra.run.smithi124.stdout:Last metadata expiration check: 0:04:02 ago on Sat 26 Aug 2023 01:25:43 PM UTC. 2023-08-26T13:29:46.376 INFO:teuthology.orchestra.run.smithi031.stdout:Installed products updated. 2023-08-26T13:29:46.570 INFO:teuthology.orchestra.run.smithi124.stdout:Dependencies resolved. 2023-08-26T13:29:46.572 INFO:teuthology.orchestra.run.smithi124.stdout:================================================================================ 2023-08-26T13:29:46.572 INFO:teuthology.orchestra.run.smithi124.stdout: Package Arch Version Repo Size 2023-08-26T13:29:46.572 INFO:teuthology.orchestra.run.smithi124.stdout:================================================================================ 2023-08-26T13:29:46.573 INFO:teuthology.orchestra.run.smithi124.stdout:Installing: 2023-08-26T13:29:46.573 INFO:teuthology.orchestra.run.smithi124.stdout: librados-devel x86_64 2:18.0.0-5706.ge7eb9853.el8 ceph 139 k 2023-08-26T13:29:46.573 INFO:teuthology.orchestra.run.smithi124.stdout: 2023-08-26T13:29:46.573 INFO:teuthology.orchestra.run.smithi124.stdout:Transaction Summary 2023-08-26T13:29:46.574 INFO:teuthology.orchestra.run.smithi124.stdout:================================================================================ 2023-08-26T13:29:46.574 INFO:teuthology.orchestra.run.smithi124.stdout:Install 1 Package 2023-08-26T13:29:46.574 INFO:teuthology.orchestra.run.smithi124.stdout: 2023-08-26T13:29:46.574 INFO:teuthology.orchestra.run.smithi124.stdout:Total download size: 139 k 2023-08-26T13:29:46.575 INFO:teuthology.orchestra.run.smithi124.stdout:Installed size: 513 k 2023-08-26T13:29:46.575 INFO:teuthology.orchestra.run.smithi124.stdout:Downloading Packages: 2023-08-26T13:29:46.576 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-08-26T13:29:46.577 INFO:teuthology.orchestra.run.smithi031.stdout:Installed: 2023-08-26T13:29:46.577 INFO:teuthology.orchestra.run.smithi031.stdout: ceph-volume-2:18.0.0-5706.ge7eb9853.el8.noarch 2023-08-26T13:29:46.577 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-08-26T13:29:46.578 INFO:teuthology.orchestra.run.smithi031.stdout:Complete! 2023-08-26T13:29:46.701 DEBUG:teuthology.orchestra.run.smithi031:> sudo yum -y install librados-devel 2023-08-26T13:29:46.750 INFO:teuthology.orchestra.run.smithi124.stdout:librados-devel-18.0.0-5706.ge7eb9853.el8.x86_64 785 kB/s | 139 kB 00:00 2023-08-26T13:29:46.751 INFO:teuthology.orchestra.run.smithi124.stdout:-------------------------------------------------------------------------------- 2023-08-26T13:29:46.751 INFO:teuthology.orchestra.run.smithi124.stdout:Total 779 kB/s | 139 kB 00:00 2023-08-26T13:29:46.751 INFO:teuthology.orchestra.run.smithi124.stdout:Running transaction check 2023-08-26T13:29:46.756 INFO:teuthology.orchestra.run.smithi124.stdout:Transaction check succeeded. 2023-08-26T13:29:46.756 INFO:teuthology.orchestra.run.smithi124.stdout:Running transaction test 2023-08-26T13:29:46.811 INFO:teuthology.orchestra.run.smithi124.stdout:Transaction test succeeded. 2023-08-26T13:29:46.813 INFO:teuthology.orchestra.run.smithi124.stdout:Running transaction 2023-08-26T13:29:46.926 INFO:teuthology.orchestra.run.smithi124.stdout: Preparing : 1/1 2023-08-26T13:29:47.080 INFO:teuthology.orchestra.run.smithi031.stdout:Updating Subscription Management repositories. 2023-08-26T13:29:47.134 INFO:teuthology.orchestra.run.smithi124.stdout: Installing : librados-devel-2:18.0.0-5706.ge7eb9853.el8.x86_64 1/1 2023-08-26T13:29:47.369 INFO:teuthology.orchestra.run.smithi124.stdout: Running scriptlet: librados-devel-2:18.0.0-5706.ge7eb9853.el8.x86_64 1/1 2023-08-26T13:29:47.952 INFO:teuthology.orchestra.run.smithi031.stdout:Last metadata expiration check: 0:04:04 ago on Sat 26 Aug 2023 01:25:43 PM UTC. 2023-08-26T13:29:48.428 INFO:teuthology.orchestra.run.smithi124.stdout: Verifying : librados-devel-2:18.0.0-5706.ge7eb9853.el8.x86_64 1/1 2023-08-26T13:29:48.428 INFO:teuthology.orchestra.run.smithi124.stdout:Last metadata expiration check: 0:04:05 ago on Sat 26 Aug 2023 01:25:43 PM UTC. 2023-08-26T13:29:49.514 INFO:teuthology.orchestra.run.smithi031.stdout:Dependencies resolved. 2023-08-26T13:29:49.515 INFO:teuthology.orchestra.run.smithi031.stdout:================================================================================ 2023-08-26T13:29:49.516 INFO:teuthology.orchestra.run.smithi031.stdout: Package Arch Version Repo Size 2023-08-26T13:29:49.516 INFO:teuthology.orchestra.run.smithi031.stdout:================================================================================ 2023-08-26T13:29:49.516 INFO:teuthology.orchestra.run.smithi031.stdout:Installing: 2023-08-26T13:29:49.516 INFO:teuthology.orchestra.run.smithi031.stdout: librados-devel x86_64 2:18.0.0-5706.ge7eb9853.el8 ceph 139 k 2023-08-26T13:29:49.516 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-08-26T13:29:49.517 INFO:teuthology.orchestra.run.smithi031.stdout:Transaction Summary 2023-08-26T13:29:49.517 INFO:teuthology.orchestra.run.smithi031.stdout:================================================================================ 2023-08-26T13:29:49.517 INFO:teuthology.orchestra.run.smithi031.stdout:Install 1 Package 2023-08-26T13:29:49.517 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-08-26T13:29:49.517 INFO:teuthology.orchestra.run.smithi031.stdout:Total download size: 139 k 2023-08-26T13:29:49.518 INFO:teuthology.orchestra.run.smithi031.stdout:Installed size: 513 k 2023-08-26T13:29:49.518 INFO:teuthology.orchestra.run.smithi031.stdout:Downloading Packages: 2023-08-26T13:29:49.795 INFO:teuthology.orchestra.run.smithi031.stdout:librados-devel-18.0.0-5706.ge7eb9853.el8.x86_64 498 kB/s | 139 kB 00:00 2023-08-26T13:29:49.796 INFO:teuthology.orchestra.run.smithi031.stdout:-------------------------------------------------------------------------------- 2023-08-26T13:29:49.796 INFO:teuthology.orchestra.run.smithi031.stdout:Total 496 kB/s | 139 kB 00:00 2023-08-26T13:29:49.797 INFO:teuthology.orchestra.run.smithi031.stdout:Running transaction check 2023-08-26T13:29:49.802 INFO:teuthology.orchestra.run.smithi031.stdout:Transaction check succeeded. 2023-08-26T13:29:49.802 INFO:teuthology.orchestra.run.smithi031.stdout:Running transaction test 2023-08-26T13:29:49.860 INFO:teuthology.orchestra.run.smithi031.stdout:Transaction test succeeded. 2023-08-26T13:29:49.864 INFO:teuthology.orchestra.run.smithi031.stdout:Running transaction 2023-08-26T13:29:49.984 INFO:teuthology.orchestra.run.smithi031.stdout: Preparing : 1/1 2023-08-26T13:29:50.136 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : librados-devel-2:18.0.0-5706.ge7eb9853.el8.x86_64 1/1 2023-08-26T13:29:50.383 INFO:teuthology.orchestra.run.smithi031.stdout: Running scriptlet: librados-devel-2:18.0.0-5706.ge7eb9853.el8.x86_64 1/1 2023-08-26T13:29:50.937 INFO:teuthology.orchestra.run.smithi124.stdout:Installed products updated. 2023-08-26T13:29:51.122 INFO:teuthology.orchestra.run.smithi124.stdout: 2023-08-26T13:29:51.123 INFO:teuthology.orchestra.run.smithi124.stdout:Installed: 2023-08-26T13:29:51.123 INFO:teuthology.orchestra.run.smithi124.stdout: librados-devel-2:18.0.0-5706.ge7eb9853.el8.x86_64 2023-08-26T13:29:51.123 INFO:teuthology.orchestra.run.smithi124.stdout: 2023-08-26T13:29:51.123 INFO:teuthology.orchestra.run.smithi124.stdout:Complete! 2023-08-26T13:29:51.259 DEBUG:teuthology.orchestra.run.smithi124:> sudo yum -y install libcephfs2 2023-08-26T13:29:51.458 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : librados-devel-2:18.0.0-5706.ge7eb9853.el8.x86_64 1/1 2023-08-26T13:29:51.458 INFO:teuthology.orchestra.run.smithi031.stdout:Last metadata expiration check: 0:04:08 ago on Sat 26 Aug 2023 01:25:43 PM UTC. 2023-08-26T13:29:51.612 INFO:teuthology.orchestra.run.smithi124.stdout:Updating Subscription Management repositories. 2023-08-26T13:29:52.425 INFO:teuthology.orchestra.run.smithi124.stdout:Last metadata expiration check: 0:04:09 ago on Sat 26 Aug 2023 01:25:43 PM UTC. 2023-08-26T13:29:53.697 INFO:teuthology.orchestra.run.smithi124.stdout:Package libcephfs2-2:18.0.0-5706.ge7eb9853.el8.x86_64 is already installed. 2023-08-26T13:29:53.804 INFO:teuthology.orchestra.run.smithi124.stdout:Dependencies resolved. 2023-08-26T13:29:53.805 INFO:teuthology.orchestra.run.smithi124.stdout:Nothing to do. 2023-08-26T13:29:53.805 INFO:teuthology.orchestra.run.smithi124.stdout:Complete! 2023-08-26T13:29:53.908 DEBUG:teuthology.orchestra.run.smithi124:> sudo yum -y install libcephfs-devel 2023-08-26T13:29:54.213 INFO:teuthology.orchestra.run.smithi031.stdout:Installed products updated. 2023-08-26T13:29:54.260 INFO:teuthology.orchestra.run.smithi124.stdout:Updating Subscription Management repositories. 2023-08-26T13:29:54.413 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-08-26T13:29:54.413 INFO:teuthology.orchestra.run.smithi031.stdout:Installed: 2023-08-26T13:29:54.413 INFO:teuthology.orchestra.run.smithi031.stdout: librados-devel-2:18.0.0-5706.ge7eb9853.el8.x86_64 2023-08-26T13:29:54.414 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-08-26T13:29:54.414 INFO:teuthology.orchestra.run.smithi031.stdout:Complete! 2023-08-26T13:29:54.673 DEBUG:teuthology.orchestra.run.smithi031:> sudo yum -y install libcephfs2 2023-08-26T13:29:55.051 INFO:teuthology.orchestra.run.smithi031.stdout:Updating Subscription Management repositories. 2023-08-26T13:29:55.101 INFO:teuthology.orchestra.run.smithi124.stdout:Last metadata expiration check: 0:04:12 ago on Sat 26 Aug 2023 01:25:43 PM UTC. 2023-08-26T13:29:55.966 INFO:teuthology.orchestra.run.smithi031.stdout:Last metadata expiration check: 0:04:12 ago on Sat 26 Aug 2023 01:25:43 PM UTC. 2023-08-26T13:29:56.479 INFO:teuthology.orchestra.run.smithi124.stdout:Dependencies resolved. 2023-08-26T13:29:56.480 INFO:teuthology.orchestra.run.smithi124.stdout:================================================================================ 2023-08-26T13:29:56.480 INFO:teuthology.orchestra.run.smithi124.stdout: Package Arch Version Repo Size 2023-08-26T13:29:56.480 INFO:teuthology.orchestra.run.smithi124.stdout:================================================================================ 2023-08-26T13:29:56.480 INFO:teuthology.orchestra.run.smithi124.stdout:Installing: 2023-08-26T13:29:56.481 INFO:teuthology.orchestra.run.smithi124.stdout: libcephfs-devel x86_64 2:18.0.0-5706.ge7eb9853.el8 ceph 31 k 2023-08-26T13:29:56.481 INFO:teuthology.orchestra.run.smithi124.stdout: 2023-08-26T13:29:56.481 INFO:teuthology.orchestra.run.smithi124.stdout:Transaction Summary 2023-08-26T13:29:56.481 INFO:teuthology.orchestra.run.smithi124.stdout:================================================================================ 2023-08-26T13:29:56.481 INFO:teuthology.orchestra.run.smithi124.stdout:Install 1 Package 2023-08-26T13:29:56.482 INFO:teuthology.orchestra.run.smithi124.stdout: 2023-08-26T13:29:56.482 INFO:teuthology.orchestra.run.smithi124.stdout:Total download size: 31 k 2023-08-26T13:29:56.482 INFO:teuthology.orchestra.run.smithi124.stdout:Installed size: 140 k 2023-08-26T13:29:56.482 INFO:teuthology.orchestra.run.smithi124.stdout:Downloading Packages: 2023-08-26T13:29:56.698 INFO:teuthology.orchestra.run.smithi124.stdout:libcephfs-devel-18.0.0-5706.ge7eb9853.el8.x86_6 143 kB/s | 31 kB 00:00 2023-08-26T13:29:56.699 INFO:teuthology.orchestra.run.smithi124.stdout:-------------------------------------------------------------------------------- 2023-08-26T13:29:56.699 INFO:teuthology.orchestra.run.smithi124.stdout:Total 142 kB/s | 31 kB 00:00 2023-08-26T13:29:56.700 INFO:teuthology.orchestra.run.smithi124.stdout:Running transaction check 2023-08-26T13:29:56.703 INFO:teuthology.orchestra.run.smithi124.stdout:Transaction check succeeded. 2023-08-26T13:29:56.703 INFO:teuthology.orchestra.run.smithi124.stdout:Running transaction test 2023-08-26T13:29:56.713 INFO:teuthology.orchestra.run.smithi124.stdout:Transaction test succeeded. 2023-08-26T13:29:56.717 INFO:teuthology.orchestra.run.smithi124.stdout:Running transaction 2023-08-26T13:29:56.771 INFO:teuthology.orchestra.run.smithi124.stdout: Preparing : 1/1 2023-08-26T13:29:56.965 INFO:teuthology.orchestra.run.smithi124.stdout: Installing : libcephfs-devel-2:18.0.0-5706.ge7eb9853.el8.x86_64 1/1 2023-08-26T13:29:57.184 INFO:teuthology.orchestra.run.smithi124.stdout: Running scriptlet: libcephfs-devel-2:18.0.0-5706.ge7eb9853.el8.x86_64 1/1 2023-08-26T13:29:57.398 INFO:teuthology.orchestra.run.smithi031.stdout:Package libcephfs2-2:18.0.0-5706.ge7eb9853.el8.x86_64 is already installed. 2023-08-26T13:29:57.520 INFO:teuthology.orchestra.run.smithi031.stdout:Dependencies resolved. 2023-08-26T13:29:57.521 INFO:teuthology.orchestra.run.smithi031.stdout:Nothing to do. 2023-08-26T13:29:57.521 INFO:teuthology.orchestra.run.smithi031.stdout:Complete! 2023-08-26T13:29:57.754 DEBUG:teuthology.orchestra.run.smithi031:> sudo yum -y install libcephfs-devel 2023-08-26T13:29:58.133 INFO:teuthology.orchestra.run.smithi031.stdout:Updating Subscription Management repositories. 2023-08-26T13:29:58.152 INFO:teuthology.orchestra.run.smithi124.stdout: Verifying : libcephfs-devel-2:18.0.0-5706.ge7eb9853.el8.x86_64 1/1 2023-08-26T13:29:58.152 INFO:teuthology.orchestra.run.smithi124.stdout:Last metadata expiration check: 0:04:15 ago on Sat 26 Aug 2023 01:25:43 PM UTC. 2023-08-26T13:29:59.014 INFO:teuthology.orchestra.run.smithi031.stdout:Last metadata expiration check: 0:04:16 ago on Sat 26 Aug 2023 01:25:43 PM UTC. 2023-08-26T13:30:00.574 INFO:teuthology.orchestra.run.smithi031.stdout:Dependencies resolved. 2023-08-26T13:30:00.575 INFO:teuthology.orchestra.run.smithi031.stdout:================================================================================ 2023-08-26T13:30:00.575 INFO:teuthology.orchestra.run.smithi031.stdout: Package Arch Version Repo Size 2023-08-26T13:30:00.575 INFO:teuthology.orchestra.run.smithi031.stdout:================================================================================ 2023-08-26T13:30:00.575 INFO:teuthology.orchestra.run.smithi031.stdout:Installing: 2023-08-26T13:30:00.575 INFO:teuthology.orchestra.run.smithi031.stdout: libcephfs-devel x86_64 2:18.0.0-5706.ge7eb9853.el8 ceph 31 k 2023-08-26T13:30:00.576 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-08-26T13:30:00.576 INFO:teuthology.orchestra.run.smithi031.stdout:Transaction Summary 2023-08-26T13:30:00.576 INFO:teuthology.orchestra.run.smithi031.stdout:================================================================================ 2023-08-26T13:30:00.576 INFO:teuthology.orchestra.run.smithi031.stdout:Install 1 Package 2023-08-26T13:30:00.576 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-08-26T13:30:00.577 INFO:teuthology.orchestra.run.smithi031.stdout:Total download size: 31 k 2023-08-26T13:30:00.577 INFO:teuthology.orchestra.run.smithi031.stdout:Installed size: 140 k 2023-08-26T13:30:00.578 INFO:teuthology.orchestra.run.smithi031.stdout:Downloading Packages: 2023-08-26T13:30:00.724 INFO:teuthology.orchestra.run.smithi031.stdout:libcephfs-devel-18.0.0-5706.ge7eb9853.el8.x86_6 209 kB/s | 31 kB 00:00 2023-08-26T13:30:00.726 INFO:teuthology.orchestra.run.smithi031.stdout:-------------------------------------------------------------------------------- 2023-08-26T13:30:00.726 INFO:teuthology.orchestra.run.smithi031.stdout:Total 208 kB/s | 31 kB 00:00 2023-08-26T13:30:00.726 INFO:teuthology.orchestra.run.smithi031.stdout:Running transaction check 2023-08-26T13:30:00.729 INFO:teuthology.orchestra.run.smithi031.stdout:Transaction check succeeded. 2023-08-26T13:30:00.730 INFO:teuthology.orchestra.run.smithi031.stdout:Running transaction test 2023-08-26T13:30:00.741 INFO:teuthology.orchestra.run.smithi031.stdout:Transaction test succeeded. 2023-08-26T13:30:00.745 INFO:teuthology.orchestra.run.smithi031.stdout:Running transaction 2023-08-26T13:30:00.754 INFO:teuthology.orchestra.run.smithi124.stdout:Installed products updated. 2023-08-26T13:30:00.806 INFO:teuthology.orchestra.run.smithi031.stdout: Preparing : 1/1 2023-08-26T13:30:00.933 INFO:teuthology.orchestra.run.smithi124.stdout: 2023-08-26T13:30:00.933 INFO:teuthology.orchestra.run.smithi124.stdout:Installed: 2023-08-26T13:30:00.934 INFO:teuthology.orchestra.run.smithi124.stdout: libcephfs-devel-2:18.0.0-5706.ge7eb9853.el8.x86_64 2023-08-26T13:30:00.934 INFO:teuthology.orchestra.run.smithi124.stdout: 2023-08-26T13:30:00.934 INFO:teuthology.orchestra.run.smithi124.stdout:Complete! 2023-08-26T13:30:00.968 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : libcephfs-devel-2:18.0.0-5706.ge7eb9853.el8.x86_64 1/1 2023-08-26T13:30:01.051 DEBUG:teuthology.orchestra.run.smithi124:> sudo yum -y install librados2 2023-08-26T13:30:01.195 INFO:teuthology.orchestra.run.smithi031.stdout: Running scriptlet: libcephfs-devel-2:18.0.0-5706.ge7eb9853.el8.x86_64 1/1 2023-08-26T13:30:01.407 INFO:teuthology.orchestra.run.smithi124.stdout:Updating Subscription Management repositories. 2023-08-26T13:30:02.215 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : libcephfs-devel-2:18.0.0-5706.ge7eb9853.el8.x86_64 1/1 2023-08-26T13:30:02.216 INFO:teuthology.orchestra.run.smithi031.stdout:Last metadata expiration check: 0:04:19 ago on Sat 26 Aug 2023 01:25:43 PM UTC. 2023-08-26T13:30:02.397 INFO:teuthology.orchestra.run.smithi124.stdout:Last metadata expiration check: 0:04:19 ago on Sat 26 Aug 2023 01:25:43 PM UTC. 2023-08-26T13:30:03.661 INFO:teuthology.orchestra.run.smithi124.stdout:Package librados2-2:18.0.0-5706.ge7eb9853.el8.x86_64 is already installed. 2023-08-26T13:30:03.770 INFO:teuthology.orchestra.run.smithi124.stdout:Dependencies resolved. 2023-08-26T13:30:03.771 INFO:teuthology.orchestra.run.smithi124.stdout:Nothing to do. 2023-08-26T13:30:03.771 INFO:teuthology.orchestra.run.smithi124.stdout:Complete! 2023-08-26T13:30:03.890 DEBUG:teuthology.orchestra.run.smithi124:> sudo yum -y install librbd1 2023-08-26T13:30:04.242 INFO:teuthology.orchestra.run.smithi124.stdout:Updating Subscription Management repositories. 2023-08-26T13:30:05.204 INFO:teuthology.orchestra.run.smithi031.stdout:Installed products updated. 2023-08-26T13:30:05.217 INFO:teuthology.orchestra.run.smithi124.stdout:Last metadata expiration check: 0:04:22 ago on Sat 26 Aug 2023 01:25:43 PM UTC. 2023-08-26T13:30:05.404 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-08-26T13:30:05.405 INFO:teuthology.orchestra.run.smithi031.stdout:Installed: 2023-08-26T13:30:05.405 INFO:teuthology.orchestra.run.smithi031.stdout: libcephfs-devel-2:18.0.0-5706.ge7eb9853.el8.x86_64 2023-08-26T13:30:05.405 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-08-26T13:30:05.405 INFO:teuthology.orchestra.run.smithi031.stdout:Complete! 2023-08-26T13:30:05.531 DEBUG:teuthology.orchestra.run.smithi031:> sudo yum -y install librados2 2023-08-26T13:30:05.908 INFO:teuthology.orchestra.run.smithi031.stdout:Updating Subscription Management repositories. 2023-08-26T13:30:06.488 INFO:teuthology.orchestra.run.smithi124.stdout:Package librbd1-2:18.0.0-5706.ge7eb9853.el8.x86_64 is already installed. 2023-08-26T13:30:06.596 INFO:teuthology.orchestra.run.smithi124.stdout:Dependencies resolved. 2023-08-26T13:30:06.597 INFO:teuthology.orchestra.run.smithi124.stdout:Nothing to do. 2023-08-26T13:30:06.597 INFO:teuthology.orchestra.run.smithi124.stdout:Complete! 2023-08-26T13:30:06.698 DEBUG:teuthology.orchestra.run.smithi124:> sudo yum -y install python3-rados 2023-08-26T13:30:06.794 INFO:teuthology.orchestra.run.smithi031.stdout:Last metadata expiration check: 0:04:23 ago on Sat 26 Aug 2023 01:25:43 PM UTC. 2023-08-26T13:30:07.052 INFO:teuthology.orchestra.run.smithi124.stdout:Updating Subscription Management repositories. 2023-08-26T13:30:08.194 INFO:teuthology.orchestra.run.smithi031.stdout:Package librados2-2:18.0.0-5706.ge7eb9853.el8.x86_64 is already installed. 2023-08-26T13:30:08.271 INFO:teuthology.orchestra.run.smithi124.stdout:Last metadata expiration check: 0:04:25 ago on Sat 26 Aug 2023 01:25:43 PM UTC. 2023-08-26T13:30:08.313 INFO:teuthology.orchestra.run.smithi031.stdout:Dependencies resolved. 2023-08-26T13:30:08.314 INFO:teuthology.orchestra.run.smithi031.stdout:Nothing to do. 2023-08-26T13:30:08.314 INFO:teuthology.orchestra.run.smithi031.stdout:Complete! 2023-08-26T13:30:08.426 DEBUG:teuthology.orchestra.run.smithi031:> sudo yum -y install librbd1 2023-08-26T13:30:08.802 INFO:teuthology.orchestra.run.smithi031.stdout:Updating Subscription Management repositories. 2023-08-26T13:30:09.559 INFO:teuthology.orchestra.run.smithi124.stdout:Package python3-rados-2:18.0.0-5706.ge7eb9853.el8.x86_64 is already installed. 2023-08-26T13:30:09.666 INFO:teuthology.orchestra.run.smithi124.stdout:Dependencies resolved. 2023-08-26T13:30:09.667 INFO:teuthology.orchestra.run.smithi124.stdout:Nothing to do. 2023-08-26T13:30:09.667 INFO:teuthology.orchestra.run.smithi124.stdout:Complete! 2023-08-26T13:30:09.772 DEBUG:teuthology.orchestra.run.smithi124:> sudo yum -y install python3-rgw 2023-08-26T13:30:09.827 INFO:teuthology.orchestra.run.smithi031.stdout:Last metadata expiration check: 0:04:26 ago on Sat 26 Aug 2023 01:25:43 PM UTC. 2023-08-26T13:30:10.124 INFO:teuthology.orchestra.run.smithi124.stdout:Updating Subscription Management repositories. 2023-08-26T13:30:10.983 INFO:teuthology.orchestra.run.smithi124.stdout:Last metadata expiration check: 0:04:27 ago on Sat 26 Aug 2023 01:25:43 PM UTC. 2023-08-26T13:30:11.249 INFO:teuthology.orchestra.run.smithi031.stdout:Package librbd1-2:18.0.0-5706.ge7eb9853.el8.x86_64 is already installed. 2023-08-26T13:30:11.364 INFO:teuthology.orchestra.run.smithi031.stdout:Dependencies resolved. 2023-08-26T13:30:11.366 INFO:teuthology.orchestra.run.smithi031.stdout:Nothing to do. 2023-08-26T13:30:11.366 INFO:teuthology.orchestra.run.smithi031.stdout:Complete! 2023-08-26T13:30:11.479 DEBUG:teuthology.orchestra.run.smithi031:> sudo yum -y install python3-rados 2023-08-26T13:30:11.856 INFO:teuthology.orchestra.run.smithi031.stdout:Updating Subscription Management repositories. 2023-08-26T13:30:12.267 INFO:teuthology.orchestra.run.smithi124.stdout:Package python3-rgw-2:18.0.0-5706.ge7eb9853.el8.x86_64 is already installed. 2023-08-26T13:30:12.372 INFO:teuthology.orchestra.run.smithi124.stdout:Dependencies resolved. 2023-08-26T13:30:12.373 INFO:teuthology.orchestra.run.smithi124.stdout:Nothing to do. 2023-08-26T13:30:12.373 INFO:teuthology.orchestra.run.smithi124.stdout:Complete! 2023-08-26T13:30:12.473 DEBUG:teuthology.orchestra.run.smithi124:> sudo yum -y install python3-cephfs 2023-08-26T13:30:12.825 INFO:teuthology.orchestra.run.smithi124.stdout:Updating Subscription Management repositories. 2023-08-26T13:30:13.508 INFO:teuthology.orchestra.run.smithi031.stdout:Last metadata expiration check: 0:04:30 ago on Sat 26 Aug 2023 01:25:43 PM UTC. 2023-08-26T13:30:14.271 INFO:teuthology.orchestra.run.smithi124.stdout:Last metadata expiration check: 0:04:31 ago on Sat 26 Aug 2023 01:25:43 PM UTC. 2023-08-26T13:30:14.946 INFO:teuthology.orchestra.run.smithi031.stdout:Package python3-rados-2:18.0.0-5706.ge7eb9853.el8.x86_64 is already installed. 2023-08-26T13:30:15.063 INFO:teuthology.orchestra.run.smithi031.stdout:Dependencies resolved. 2023-08-26T13:30:15.065 INFO:teuthology.orchestra.run.smithi031.stdout:Nothing to do. 2023-08-26T13:30:15.065 INFO:teuthology.orchestra.run.smithi031.stdout:Complete! 2023-08-26T13:30:15.176 DEBUG:teuthology.orchestra.run.smithi031:> sudo yum -y install python3-rgw 2023-08-26T13:30:15.552 INFO:teuthology.orchestra.run.smithi031.stdout:Updating Subscription Management repositories. 2023-08-26T13:30:15.610 INFO:teuthology.orchestra.run.smithi124.stdout:Package python3-cephfs-2:18.0.0-5706.ge7eb9853.el8.x86_64 is already installed. 2023-08-26T13:30:15.719 INFO:teuthology.orchestra.run.smithi124.stdout:Dependencies resolved. 2023-08-26T13:30:15.720 INFO:teuthology.orchestra.run.smithi124.stdout:Nothing to do. 2023-08-26T13:30:15.720 INFO:teuthology.orchestra.run.smithi124.stdout:Complete! 2023-08-26T13:30:15.822 DEBUG:teuthology.orchestra.run.smithi124:> sudo yum -y install python3-rbd 2023-08-26T13:30:16.174 INFO:teuthology.orchestra.run.smithi124.stdout:Updating Subscription Management repositories. 2023-08-26T13:30:16.967 INFO:teuthology.orchestra.run.smithi031.stdout:Last metadata expiration check: 0:04:33 ago on Sat 26 Aug 2023 01:25:43 PM UTC. 2023-08-26T13:30:17.436 INFO:teuthology.orchestra.run.smithi124.stdout:Last metadata expiration check: 0:04:34 ago on Sat 26 Aug 2023 01:25:43 PM UTC. 2023-08-26T13:30:18.436 INFO:teuthology.orchestra.run.smithi031.stdout:Package python3-rgw-2:18.0.0-5706.ge7eb9853.el8.x86_64 is already installed. 2023-08-26T13:30:18.558 INFO:teuthology.orchestra.run.smithi031.stdout:Dependencies resolved. 2023-08-26T13:30:18.559 INFO:teuthology.orchestra.run.smithi031.stdout:Nothing to do. 2023-08-26T13:30:18.559 INFO:teuthology.orchestra.run.smithi031.stdout:Complete! 2023-08-26T13:30:18.674 DEBUG:teuthology.orchestra.run.smithi031:> sudo yum -y install python3-cephfs 2023-08-26T13:30:18.692 INFO:teuthology.orchestra.run.smithi124.stdout:Package python3-rbd-2:18.0.0-5706.ge7eb9853.el8.x86_64 is already installed. 2023-08-26T13:30:18.802 INFO:teuthology.orchestra.run.smithi124.stdout:Dependencies resolved. 2023-08-26T13:30:18.803 INFO:teuthology.orchestra.run.smithi124.stdout:Nothing to do. 2023-08-26T13:30:18.804 INFO:teuthology.orchestra.run.smithi124.stdout:Complete! 2023-08-26T13:30:18.904 DEBUG:teuthology.orchestra.run.smithi124:> sudo yum -y install rbd-fuse 2023-08-26T13:30:19.049 INFO:teuthology.orchestra.run.smithi031.stdout:Updating Subscription Management repositories. 2023-08-26T13:30:19.257 INFO:teuthology.orchestra.run.smithi124.stdout:Updating Subscription Management repositories. 2023-08-26T13:30:19.935 INFO:teuthology.orchestra.run.smithi031.stdout:Last metadata expiration check: 0:04:36 ago on Sat 26 Aug 2023 01:25:43 PM UTC. 2023-08-26T13:30:20.141 INFO:teuthology.orchestra.run.smithi124.stdout:Last metadata expiration check: 0:04:37 ago on Sat 26 Aug 2023 01:25:43 PM UTC. 2023-08-26T13:30:21.388 INFO:teuthology.orchestra.run.smithi031.stdout:Package python3-cephfs-2:18.0.0-5706.ge7eb9853.el8.x86_64 is already installed. 2023-08-26T13:30:21.504 INFO:teuthology.orchestra.run.smithi031.stdout:Dependencies resolved. 2023-08-26T13:30:21.505 INFO:teuthology.orchestra.run.smithi031.stdout:Nothing to do. 2023-08-26T13:30:21.505 INFO:teuthology.orchestra.run.smithi031.stdout:Complete! 2023-08-26T13:30:21.527 INFO:teuthology.orchestra.run.smithi124.stdout:Dependencies resolved. 2023-08-26T13:30:21.527 INFO:teuthology.orchestra.run.smithi124.stdout:================================================================================ 2023-08-26T13:30:21.527 INFO:teuthology.orchestra.run.smithi124.stdout: Package Architecture Version Repository Size 2023-08-26T13:30:21.528 INFO:teuthology.orchestra.run.smithi124.stdout:================================================================================ 2023-08-26T13:30:21.528 INFO:teuthology.orchestra.run.smithi124.stdout:Installing: 2023-08-26T13:30:21.528 INFO:teuthology.orchestra.run.smithi124.stdout: rbd-fuse x86_64 2:18.0.0-5706.ge7eb9853.el8 ceph 93 k 2023-08-26T13:30:21.528 INFO:teuthology.orchestra.run.smithi124.stdout: 2023-08-26T13:30:21.528 INFO:teuthology.orchestra.run.smithi124.stdout:Transaction Summary 2023-08-26T13:30:21.529 INFO:teuthology.orchestra.run.smithi124.stdout:================================================================================ 2023-08-26T13:30:21.529 INFO:teuthology.orchestra.run.smithi124.stdout:Install 1 Package 2023-08-26T13:30:21.529 INFO:teuthology.orchestra.run.smithi124.stdout: 2023-08-26T13:30:21.529 INFO:teuthology.orchestra.run.smithi124.stdout:Total download size: 93 k 2023-08-26T13:30:21.529 INFO:teuthology.orchestra.run.smithi124.stdout:Installed size: 276 k 2023-08-26T13:30:21.529 INFO:teuthology.orchestra.run.smithi124.stdout:Downloading Packages: 2023-08-26T13:30:21.619 DEBUG:teuthology.orchestra.run.smithi031:> sudo yum -y install python3-rbd 2023-08-26T13:30:21.819 INFO:teuthology.orchestra.run.smithi124.stdout:rbd-fuse-18.0.0-5706.ge7eb9853.el8.x86_64.rpm 321 kB/s | 93 kB 00:00 2023-08-26T13:30:21.820 INFO:teuthology.orchestra.run.smithi124.stdout:-------------------------------------------------------------------------------- 2023-08-26T13:30:21.820 INFO:teuthology.orchestra.run.smithi124.stdout:Total 320 kB/s | 93 kB 00:00 2023-08-26T13:30:21.821 INFO:teuthology.orchestra.run.smithi124.stdout:Running transaction check 2023-08-26T13:30:21.830 INFO:teuthology.orchestra.run.smithi124.stdout:Transaction check succeeded. 2023-08-26T13:30:21.830 INFO:teuthology.orchestra.run.smithi124.stdout:Running transaction test 2023-08-26T13:30:21.884 INFO:teuthology.orchestra.run.smithi124.stdout:Transaction test succeeded. 2023-08-26T13:30:21.887 INFO:teuthology.orchestra.run.smithi124.stdout:Running transaction 2023-08-26T13:30:21.996 INFO:teuthology.orchestra.run.smithi031.stdout:Updating Subscription Management repositories. 2023-08-26T13:30:22.001 INFO:teuthology.orchestra.run.smithi124.stdout: Preparing : 1/1 2023-08-26T13:30:22.163 INFO:teuthology.orchestra.run.smithi124.stdout: Installing : rbd-fuse-2:18.0.0-5706.ge7eb9853.el8.x86_64 1/1 2023-08-26T13:30:22.403 INFO:teuthology.orchestra.run.smithi124.stdout: Running scriptlet: rbd-fuse-2:18.0.0-5706.ge7eb9853.el8.x86_64 1/1 2023-08-26T13:30:22.865 INFO:teuthology.orchestra.run.smithi031.stdout:Last metadata expiration check: 0:04:39 ago on Sat 26 Aug 2023 01:25:43 PM UTC. 2023-08-26T13:30:23.474 INFO:teuthology.orchestra.run.smithi124.stdout: Verifying : rbd-fuse-2:18.0.0-5706.ge7eb9853.el8.x86_64 1/1 2023-08-26T13:30:23.474 INFO:teuthology.orchestra.run.smithi124.stdout:Last metadata expiration check: 0:04:40 ago on Sat 26 Aug 2023 01:25:43 PM UTC. 2023-08-26T13:30:24.307 INFO:teuthology.orchestra.run.smithi031.stdout:Package python3-rbd-2:18.0.0-5706.ge7eb9853.el8.x86_64 is already installed. 2023-08-26T13:30:24.426 INFO:teuthology.orchestra.run.smithi031.stdout:Dependencies resolved. 2023-08-26T13:30:24.427 INFO:teuthology.orchestra.run.smithi031.stdout:Nothing to do. 2023-08-26T13:30:24.428 INFO:teuthology.orchestra.run.smithi031.stdout:Complete! 2023-08-26T13:30:24.540 DEBUG:teuthology.orchestra.run.smithi031:> sudo yum -y install rbd-fuse 2023-08-26T13:30:24.916 INFO:teuthology.orchestra.run.smithi031.stdout:Updating Subscription Management repositories. 2023-08-26T13:30:25.882 INFO:teuthology.orchestra.run.smithi031.stdout:Last metadata expiration check: 0:04:42 ago on Sat 26 Aug 2023 01:25:43 PM UTC. 2023-08-26T13:30:25.916 INFO:teuthology.orchestra.run.smithi124.stdout:Installed products updated. 2023-08-26T13:30:26.098 INFO:teuthology.orchestra.run.smithi124.stdout: 2023-08-26T13:30:26.099 INFO:teuthology.orchestra.run.smithi124.stdout:Installed: 2023-08-26T13:30:26.099 INFO:teuthology.orchestra.run.smithi124.stdout: rbd-fuse-2:18.0.0-5706.ge7eb9853.el8.x86_64 2023-08-26T13:30:26.099 INFO:teuthology.orchestra.run.smithi124.stdout: 2023-08-26T13:30:26.099 INFO:teuthology.orchestra.run.smithi124.stdout:Complete! 2023-08-26T13:30:26.228 DEBUG:teuthology.orchestra.run.smithi124:> sudo yum -y install rbd-mirror 2023-08-26T13:30:26.581 INFO:teuthology.orchestra.run.smithi124.stdout:Updating Subscription Management repositories. 2023-08-26T13:30:27.398 INFO:teuthology.orchestra.run.smithi124.stdout:Last metadata expiration check: 0:04:44 ago on Sat 26 Aug 2023 01:25:43 PM UTC. 2023-08-26T13:30:27.457 INFO:teuthology.orchestra.run.smithi031.stdout:Dependencies resolved. 2023-08-26T13:30:27.458 INFO:teuthology.orchestra.run.smithi031.stdout:================================================================================ 2023-08-26T13:30:27.458 INFO:teuthology.orchestra.run.smithi031.stdout: Package Architecture Version Repository Size 2023-08-26T13:30:27.459 INFO:teuthology.orchestra.run.smithi031.stdout:================================================================================ 2023-08-26T13:30:27.459 INFO:teuthology.orchestra.run.smithi031.stdout:Installing: 2023-08-26T13:30:27.459 INFO:teuthology.orchestra.run.smithi031.stdout: rbd-fuse x86_64 2:18.0.0-5706.ge7eb9853.el8 ceph 93 k 2023-08-26T13:30:27.459 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-08-26T13:30:27.459 INFO:teuthology.orchestra.run.smithi031.stdout:Transaction Summary 2023-08-26T13:30:27.460 INFO:teuthology.orchestra.run.smithi031.stdout:================================================================================ 2023-08-26T13:30:27.460 INFO:teuthology.orchestra.run.smithi031.stdout:Install 1 Package 2023-08-26T13:30:27.460 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-08-26T13:30:27.460 INFO:teuthology.orchestra.run.smithi031.stdout:Total download size: 93 k 2023-08-26T13:30:27.460 INFO:teuthology.orchestra.run.smithi031.stdout:Installed size: 276 k 2023-08-26T13:30:27.461 INFO:teuthology.orchestra.run.smithi031.stdout:Downloading Packages: 2023-08-26T13:30:27.618 INFO:teuthology.orchestra.run.smithi031.stdout:rbd-fuse-18.0.0-5706.ge7eb9853.el8.x86_64.rpm 589 kB/s | 93 kB 00:00 2023-08-26T13:30:27.619 INFO:teuthology.orchestra.run.smithi031.stdout:-------------------------------------------------------------------------------- 2023-08-26T13:30:27.619 INFO:teuthology.orchestra.run.smithi031.stdout:Total 584 kB/s | 93 kB 00:00 2023-08-26T13:30:27.619 INFO:teuthology.orchestra.run.smithi031.stdout:Running transaction check 2023-08-26T13:30:27.631 INFO:teuthology.orchestra.run.smithi031.stdout:Transaction check succeeded. 2023-08-26T13:30:27.631 INFO:teuthology.orchestra.run.smithi031.stdout:Running transaction test 2023-08-26T13:30:27.688 INFO:teuthology.orchestra.run.smithi031.stdout:Transaction test succeeded. 2023-08-26T13:30:27.692 INFO:teuthology.orchestra.run.smithi031.stdout:Running transaction 2023-08-26T13:30:27.805 INFO:teuthology.orchestra.run.smithi031.stdout: Preparing : 1/1 2023-08-26T13:30:27.947 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : rbd-fuse-2:18.0.0-5706.ge7eb9853.el8.x86_64 1/1 2023-08-26T13:30:28.171 INFO:teuthology.orchestra.run.smithi031.stdout: Running scriptlet: rbd-fuse-2:18.0.0-5706.ge7eb9853.el8.x86_64 1/1 2023-08-26T13:30:28.770 INFO:teuthology.orchestra.run.smithi124.stdout:Dependencies resolved. 2023-08-26T13:30:28.771 INFO:teuthology.orchestra.run.smithi124.stdout:================================================================================ 2023-08-26T13:30:28.771 INFO:teuthology.orchestra.run.smithi124.stdout: Package Arch Version Repo Size 2023-08-26T13:30:28.772 INFO:teuthology.orchestra.run.smithi124.stdout:================================================================================ 2023-08-26T13:30:28.772 INFO:teuthology.orchestra.run.smithi124.stdout:Installing: 2023-08-26T13:30:28.772 INFO:teuthology.orchestra.run.smithi124.stdout: rbd-mirror x86_64 2:18.0.0-5706.ge7eb9853.el8 ceph 3.5 M 2023-08-26T13:30:28.772 INFO:teuthology.orchestra.run.smithi124.stdout: 2023-08-26T13:30:28.772 INFO:teuthology.orchestra.run.smithi124.stdout:Transaction Summary 2023-08-26T13:30:28.773 INFO:teuthology.orchestra.run.smithi124.stdout:================================================================================ 2023-08-26T13:30:28.773 INFO:teuthology.orchestra.run.smithi124.stdout:Install 1 Package 2023-08-26T13:30:28.773 INFO:teuthology.orchestra.run.smithi124.stdout: 2023-08-26T13:30:28.773 INFO:teuthology.orchestra.run.smithi124.stdout:Total download size: 3.5 M 2023-08-26T13:30:28.773 INFO:teuthology.orchestra.run.smithi124.stdout:Installed size: 17 M 2023-08-26T13:30:28.774 INFO:teuthology.orchestra.run.smithi124.stdout:Downloading Packages: 2023-08-26T13:30:29.293 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : rbd-fuse-2:18.0.0-5706.ge7eb9853.el8.x86_64 1/1 2023-08-26T13:30:29.293 INFO:teuthology.orchestra.run.smithi031.stdout:Last metadata expiration check: 0:04:46 ago on Sat 26 Aug 2023 01:25:43 PM UTC. 2023-08-26T13:30:29.358 INFO:teuthology.orchestra.run.smithi124.stdout:rbd-mirror-18.0.0-5706.ge7eb9853.el8.x86_64.rpm 6.0 MB/s | 3.5 MB 00:00 2023-08-26T13:30:29.358 INFO:teuthology.orchestra.run.smithi124.stdout:-------------------------------------------------------------------------------- 2023-08-26T13:30:29.359 INFO:teuthology.orchestra.run.smithi124.stdout:Total 6.0 MB/s | 3.5 MB 00:00 2023-08-26T13:30:29.359 INFO:teuthology.orchestra.run.smithi124.stdout:Running transaction check 2023-08-26T13:30:29.371 INFO:teuthology.orchestra.run.smithi124.stdout:Transaction check succeeded. 2023-08-26T13:30:29.371 INFO:teuthology.orchestra.run.smithi124.stdout:Running transaction test 2023-08-26T13:30:29.437 INFO:teuthology.orchestra.run.smithi124.stdout:Transaction test succeeded. 2023-08-26T13:30:29.441 INFO:teuthology.orchestra.run.smithi124.stdout:Running transaction 2023-08-26T13:30:29.903 INFO:teuthology.orchestra.run.smithi124.stdout: Preparing : 1/1 2023-08-26T13:30:30.072 INFO:teuthology.orchestra.run.smithi124.stdout: Installing : rbd-mirror-2:18.0.0-5706.ge7eb9853.el8.x86_64 1/1 2023-08-26T13:30:30.424 INFO:teuthology.orchestra.run.smithi124.stdout: Running scriptlet: rbd-mirror-2:18.0.0-5706.ge7eb9853.el8.x86_64 1/1 2023-08-26T13:30:31.521 INFO:teuthology.orchestra.run.smithi124.stdout: Verifying : rbd-mirror-2:18.0.0-5706.ge7eb9853.el8.x86_64 1/1 2023-08-26T13:30:31.522 INFO:teuthology.orchestra.run.smithi124.stdout:Last metadata expiration check: 0:04:48 ago on Sat 26 Aug 2023 01:25:43 PM UTC. 2023-08-26T13:30:31.851 INFO:teuthology.orchestra.run.smithi031.stdout:Installed products updated. 2023-08-26T13:30:32.050 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-08-26T13:30:32.050 INFO:teuthology.orchestra.run.smithi031.stdout:Installed: 2023-08-26T13:30:32.051 INFO:teuthology.orchestra.run.smithi031.stdout: rbd-fuse-2:18.0.0-5706.ge7eb9853.el8.x86_64 2023-08-26T13:30:32.051 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-08-26T13:30:32.051 INFO:teuthology.orchestra.run.smithi031.stdout:Complete! 2023-08-26T13:30:32.187 DEBUG:teuthology.orchestra.run.smithi031:> sudo yum -y install rbd-mirror 2023-08-26T13:30:32.563 INFO:teuthology.orchestra.run.smithi031.stdout:Updating Subscription Management repositories. 2023-08-26T13:30:33.444 INFO:teuthology.orchestra.run.smithi031.stdout:Last metadata expiration check: 0:04:50 ago on Sat 26 Aug 2023 01:25:43 PM UTC. 2023-08-26T13:30:34.776 INFO:teuthology.orchestra.run.smithi124.stdout:Installed products updated. 2023-08-26T13:30:34.958 INFO:teuthology.orchestra.run.smithi124.stdout: 2023-08-26T13:30:34.958 INFO:teuthology.orchestra.run.smithi124.stdout:Installed: 2023-08-26T13:30:34.958 INFO:teuthology.orchestra.run.smithi124.stdout: rbd-mirror-2:18.0.0-5706.ge7eb9853.el8.x86_64 2023-08-26T13:30:34.958 INFO:teuthology.orchestra.run.smithi124.stdout: 2023-08-26T13:30:34.958 INFO:teuthology.orchestra.run.smithi124.stdout:Complete! 2023-08-26T13:30:34.991 INFO:teuthology.orchestra.run.smithi031.stdout:Dependencies resolved. 2023-08-26T13:30:34.992 INFO:teuthology.orchestra.run.smithi031.stdout:================================================================================ 2023-08-26T13:30:34.992 INFO:teuthology.orchestra.run.smithi031.stdout: Package Arch Version Repo Size 2023-08-26T13:30:34.992 INFO:teuthology.orchestra.run.smithi031.stdout:================================================================================ 2023-08-26T13:30:34.992 INFO:teuthology.orchestra.run.smithi031.stdout:Installing: 2023-08-26T13:30:34.992 INFO:teuthology.orchestra.run.smithi031.stdout: rbd-mirror x86_64 2:18.0.0-5706.ge7eb9853.el8 ceph 3.5 M 2023-08-26T13:30:34.993 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-08-26T13:30:34.993 INFO:teuthology.orchestra.run.smithi031.stdout:Transaction Summary 2023-08-26T13:30:34.993 INFO:teuthology.orchestra.run.smithi031.stdout:================================================================================ 2023-08-26T13:30:34.993 INFO:teuthology.orchestra.run.smithi031.stdout:Install 1 Package 2023-08-26T13:30:34.993 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-08-26T13:30:34.994 INFO:teuthology.orchestra.run.smithi031.stdout:Total download size: 3.5 M 2023-08-26T13:30:34.994 INFO:teuthology.orchestra.run.smithi031.stdout:Installed size: 17 M 2023-08-26T13:30:34.994 INFO:teuthology.orchestra.run.smithi031.stdout:Downloading Packages: 2023-08-26T13:30:35.069 DEBUG:teuthology.orchestra.run.smithi124:> sudo yum -y install rbd-nbd 2023-08-26T13:30:35.424 INFO:teuthology.orchestra.run.smithi124.stdout:Updating Subscription Management repositories. 2023-08-26T13:30:35.537 INFO:teuthology.orchestra.run.smithi031.stdout:rbd-mirror-18.0.0-5706.ge7eb9853.el8.x86_64.rpm 6.5 MB/s | 3.5 MB 00:00 2023-08-26T13:30:35.538 INFO:teuthology.orchestra.run.smithi031.stdout:-------------------------------------------------------------------------------- 2023-08-26T13:30:35.539 INFO:teuthology.orchestra.run.smithi031.stdout:Total 6.5 MB/s | 3.5 MB 00:00 2023-08-26T13:30:35.539 INFO:teuthology.orchestra.run.smithi031.stdout:Running transaction check 2023-08-26T13:30:35.552 INFO:teuthology.orchestra.run.smithi031.stdout:Transaction check succeeded. 2023-08-26T13:30:35.553 INFO:teuthology.orchestra.run.smithi031.stdout:Running transaction test 2023-08-26T13:30:35.623 INFO:teuthology.orchestra.run.smithi031.stdout:Transaction test succeeded. 2023-08-26T13:30:35.627 INFO:teuthology.orchestra.run.smithi031.stdout:Running transaction 2023-08-26T13:30:36.102 INFO:teuthology.orchestra.run.smithi031.stdout: Preparing : 1/1 2023-08-26T13:30:36.257 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : rbd-mirror-2:18.0.0-5706.ge7eb9853.el8.x86_64 1/1 2023-08-26T13:30:36.370 INFO:teuthology.orchestra.run.smithi124.stdout:Last metadata expiration check: 0:04:53 ago on Sat 26 Aug 2023 01:25:43 PM UTC. 2023-08-26T13:30:36.620 INFO:teuthology.orchestra.run.smithi031.stdout: Running scriptlet: rbd-mirror-2:18.0.0-5706.ge7eb9853.el8.x86_64 1/1 2023-08-26T13:30:37.675 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : rbd-mirror-2:18.0.0-5706.ge7eb9853.el8.x86_64 1/1 2023-08-26T13:30:37.676 INFO:teuthology.orchestra.run.smithi031.stdout:Last metadata expiration check: 0:04:54 ago on Sat 26 Aug 2023 01:25:43 PM UTC. 2023-08-26T13:30:37.758 INFO:teuthology.orchestra.run.smithi124.stdout:Dependencies resolved. 2023-08-26T13:30:37.759 INFO:teuthology.orchestra.run.smithi124.stdout:================================================================================ 2023-08-26T13:30:37.759 INFO:teuthology.orchestra.run.smithi124.stdout: Package Architecture Version Repository Size 2023-08-26T13:30:37.759 INFO:teuthology.orchestra.run.smithi124.stdout:================================================================================ 2023-08-26T13:30:37.759 INFO:teuthology.orchestra.run.smithi124.stdout:Installing: 2023-08-26T13:30:37.759 INFO:teuthology.orchestra.run.smithi124.stdout: rbd-nbd x86_64 2:18.0.0-5706.ge7eb9853.el8 ceph 179 k 2023-08-26T13:30:37.759 INFO:teuthology.orchestra.run.smithi124.stdout: 2023-08-26T13:30:37.760 INFO:teuthology.orchestra.run.smithi124.stdout:Transaction Summary 2023-08-26T13:30:37.760 INFO:teuthology.orchestra.run.smithi124.stdout:================================================================================ 2023-08-26T13:30:37.760 INFO:teuthology.orchestra.run.smithi124.stdout:Install 1 Package 2023-08-26T13:30:37.760 INFO:teuthology.orchestra.run.smithi124.stdout: 2023-08-26T13:30:37.760 INFO:teuthology.orchestra.run.smithi124.stdout:Total download size: 179 k 2023-08-26T13:30:37.761 INFO:teuthology.orchestra.run.smithi124.stdout:Installed size: 565 k 2023-08-26T13:30:37.761 INFO:teuthology.orchestra.run.smithi124.stdout:Downloading Packages: 2023-08-26T13:30:38.047 INFO:teuthology.orchestra.run.smithi124.stdout:rbd-nbd-18.0.0-5706.ge7eb9853.el8.x86_64.rpm 623 kB/s | 179 kB 00:00 2023-08-26T13:30:38.048 INFO:teuthology.orchestra.run.smithi124.stdout:-------------------------------------------------------------------------------- 2023-08-26T13:30:38.048 INFO:teuthology.orchestra.run.smithi124.stdout:Total 621 kB/s | 179 kB 00:00 2023-08-26T13:30:38.048 INFO:teuthology.orchestra.run.smithi124.stdout:Running transaction check 2023-08-26T13:30:38.059 INFO:teuthology.orchestra.run.smithi124.stdout:Transaction check succeeded. 2023-08-26T13:30:38.060 INFO:teuthology.orchestra.run.smithi124.stdout:Running transaction test 2023-08-26T13:30:38.115 INFO:teuthology.orchestra.run.smithi124.stdout:Transaction test succeeded. 2023-08-26T13:30:38.118 INFO:teuthology.orchestra.run.smithi124.stdout:Running transaction 2023-08-26T13:30:38.234 INFO:teuthology.orchestra.run.smithi124.stdout: Preparing : 1/1 2023-08-26T13:30:38.393 INFO:teuthology.orchestra.run.smithi124.stdout: Installing : rbd-nbd-2:18.0.0-5706.ge7eb9853.el8.x86_64 1/1 2023-08-26T13:30:38.626 INFO:teuthology.orchestra.run.smithi124.stdout: Running scriptlet: rbd-nbd-2:18.0.0-5706.ge7eb9853.el8.x86_64 1/1 2023-08-26T13:30:39.632 INFO:teuthology.orchestra.run.smithi124.stdout: Verifying : rbd-nbd-2:18.0.0-5706.ge7eb9853.el8.x86_64 1/1 2023-08-26T13:30:39.632 INFO:teuthology.orchestra.run.smithi124.stdout:Last metadata expiration check: 0:04:56 ago on Sat 26 Aug 2023 01:25:43 PM UTC. 2023-08-26T13:30:40.432 INFO:teuthology.orchestra.run.smithi031.stdout:Installed products updated. 2023-08-26T13:30:40.631 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-08-26T13:30:40.632 INFO:teuthology.orchestra.run.smithi031.stdout:Installed: 2023-08-26T13:30:40.632 INFO:teuthology.orchestra.run.smithi031.stdout: rbd-mirror-2:18.0.0-5706.ge7eb9853.el8.x86_64 2023-08-26T13:30:40.632 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-08-26T13:30:40.632 INFO:teuthology.orchestra.run.smithi031.stdout:Complete! 2023-08-26T13:30:40.759 DEBUG:teuthology.orchestra.run.smithi031:> sudo yum -y install rbd-nbd 2023-08-26T13:30:41.139 INFO:teuthology.orchestra.run.smithi031.stdout:Updating Subscription Management repositories. 2023-08-26T13:30:42.075 INFO:teuthology.orchestra.run.smithi031.stdout:Last metadata expiration check: 0:04:59 ago on Sat 26 Aug 2023 01:25:43 PM UTC. 2023-08-26T13:30:42.371 INFO:teuthology.orchestra.run.smithi124.stdout:Installed products updated. 2023-08-26T13:30:42.553 INFO:teuthology.orchestra.run.smithi124.stdout: 2023-08-26T13:30:42.554 INFO:teuthology.orchestra.run.smithi124.stdout:Installed: 2023-08-26T13:30:42.554 INFO:teuthology.orchestra.run.smithi124.stdout: rbd-nbd-2:18.0.0-5706.ge7eb9853.el8.x86_64 2023-08-26T13:30:42.554 INFO:teuthology.orchestra.run.smithi124.stdout: 2023-08-26T13:30:42.555 INFO:teuthology.orchestra.run.smithi124.stdout:Complete! 2023-08-26T13:30:42.705 DEBUG:teuthology.orchestra.run.smithi124:> sudo yum -y install rbd-mirror 2023-08-26T13:30:43.059 INFO:teuthology.orchestra.run.smithi124.stdout:Updating Subscription Management repositories. 2023-08-26T13:30:43.564 INFO:teuthology.orchestra.run.smithi031.stdout:Dependencies resolved. 2023-08-26T13:30:43.565 INFO:teuthology.orchestra.run.smithi031.stdout:================================================================================ 2023-08-26T13:30:43.565 INFO:teuthology.orchestra.run.smithi031.stdout: Package Architecture Version Repository Size 2023-08-26T13:30:43.565 INFO:teuthology.orchestra.run.smithi031.stdout:================================================================================ 2023-08-26T13:30:43.565 INFO:teuthology.orchestra.run.smithi031.stdout:Installing: 2023-08-26T13:30:43.566 INFO:teuthology.orchestra.run.smithi031.stdout: rbd-nbd x86_64 2:18.0.0-5706.ge7eb9853.el8 ceph 179 k 2023-08-26T13:30:43.566 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-08-26T13:30:43.566 INFO:teuthology.orchestra.run.smithi031.stdout:Transaction Summary 2023-08-26T13:30:43.566 INFO:teuthology.orchestra.run.smithi031.stdout:================================================================================ 2023-08-26T13:30:43.566 INFO:teuthology.orchestra.run.smithi031.stdout:Install 1 Package 2023-08-26T13:30:43.567 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-08-26T13:30:43.567 INFO:teuthology.orchestra.run.smithi031.stdout:Total download size: 179 k 2023-08-26T13:30:43.567 INFO:teuthology.orchestra.run.smithi031.stdout:Installed size: 565 k 2023-08-26T13:30:43.567 INFO:teuthology.orchestra.run.smithi031.stdout:Downloading Packages: 2023-08-26T13:30:43.740 INFO:teuthology.orchestra.run.smithi031.stdout:rbd-nbd-18.0.0-5706.ge7eb9853.el8.x86_64.rpm 1.0 MB/s | 179 kB 00:00 2023-08-26T13:30:43.741 INFO:teuthology.orchestra.run.smithi031.stdout:-------------------------------------------------------------------------------- 2023-08-26T13:30:43.741 INFO:teuthology.orchestra.run.smithi031.stdout:Total 1.0 MB/s | 179 kB 00:00 2023-08-26T13:30:43.741 INFO:teuthology.orchestra.run.smithi031.stdout:Running transaction check 2023-08-26T13:30:43.754 INFO:teuthology.orchestra.run.smithi031.stdout:Transaction check succeeded. 2023-08-26T13:30:43.754 INFO:teuthology.orchestra.run.smithi031.stdout:Running transaction test 2023-08-26T13:30:43.812 INFO:teuthology.orchestra.run.smithi031.stdout:Transaction test succeeded. 2023-08-26T13:30:43.816 INFO:teuthology.orchestra.run.smithi031.stdout:Running transaction 2023-08-26T13:30:43.890 INFO:teuthology.orchestra.run.smithi124.stdout:Last metadata expiration check: 0:05:00 ago on Sat 26 Aug 2023 01:25:43 PM UTC. 2023-08-26T13:30:43.946 INFO:teuthology.orchestra.run.smithi031.stdout: Preparing : 1/1 2023-08-26T13:30:44.086 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : rbd-nbd-2:18.0.0-5706.ge7eb9853.el8.x86_64 1/1 2023-08-26T13:30:44.330 INFO:teuthology.orchestra.run.smithi031.stdout: Running scriptlet: rbd-nbd-2:18.0.0-5706.ge7eb9853.el8.x86_64 1/1 2023-08-26T13:30:45.161 INFO:teuthology.orchestra.run.smithi124.stdout:Package rbd-mirror-2:18.0.0-5706.ge7eb9853.el8.x86_64 is already installed. 2023-08-26T13:30:45.269 INFO:teuthology.orchestra.run.smithi124.stdout:Dependencies resolved. 2023-08-26T13:30:45.270 INFO:teuthology.orchestra.run.smithi124.stdout:Nothing to do. 2023-08-26T13:30:45.271 INFO:teuthology.orchestra.run.smithi124.stdout:Complete! 2023-08-26T13:30:45.368 DEBUG:teuthology.parallel:result is None 2023-08-26T13:30:45.448 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : rbd-nbd-2:18.0.0-5706.ge7eb9853.el8.x86_64 1/1 2023-08-26T13:30:45.448 INFO:teuthology.orchestra.run.smithi031.stdout:Last metadata expiration check: 0:05:02 ago on Sat 26 Aug 2023 01:25:43 PM UTC. 2023-08-26T13:30:49.749 INFO:teuthology.orchestra.run.smithi031.stdout:Installed products updated. 2023-08-26T13:30:49.950 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-08-26T13:30:49.950 INFO:teuthology.orchestra.run.smithi031.stdout:Installed: 2023-08-26T13:30:49.950 INFO:teuthology.orchestra.run.smithi031.stdout: rbd-nbd-2:18.0.0-5706.ge7eb9853.el8.x86_64 2023-08-26T13:30:49.950 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-08-26T13:30:49.951 INFO:teuthology.orchestra.run.smithi031.stdout:Complete! 2023-08-26T13:30:50.079 DEBUG:teuthology.orchestra.run.smithi031:> sudo yum -y install rbd-mirror 2023-08-26T13:30:50.453 INFO:teuthology.orchestra.run.smithi031.stdout:Updating Subscription Management repositories. 2023-08-26T13:30:51.270 INFO:teuthology.orchestra.run.smithi031.stdout:Last metadata expiration check: 0:05:08 ago on Sat 26 Aug 2023 01:25:43 PM UTC. 2023-08-26T13:30:52.639 INFO:teuthology.orchestra.run.smithi031.stdout:Package rbd-mirror-2:18.0.0-5706.ge7eb9853.el8.x86_64 is already installed. 2023-08-26T13:30:52.757 INFO:teuthology.orchestra.run.smithi031.stdout:Dependencies resolved. 2023-08-26T13:30:52.758 INFO:teuthology.orchestra.run.smithi031.stdout:Nothing to do. 2023-08-26T13:30:52.758 INFO:teuthology.orchestra.run.smithi031.stdout:Complete! 2023-08-26T13:30:52.877 DEBUG:teuthology.parallel:result is None 2023-08-26T13:30:52.878 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&sha1=e7eb9853fddaf661c68f207346131a7e00d8ed2f 2023-08-26T13:30:53.109 DEBUG:teuthology.orchestra.run.smithi031:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2023-08-26T13:30:53.132 INFO:teuthology.orchestra.run.smithi031.stdout:18.0.0-5706.ge7eb9853.el8 2023-08-26T13:30:53.133 INFO:teuthology.packaging:The installed version of ceph is 18.0.0-5706.ge7eb9853.el8 2023-08-26T13:30:53.134 INFO:teuthology.task.install:The correct ceph version 18.0.0-5706.ge7eb9853 is installed. 2023-08-26T13:30:53.134 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&sha1=e7eb9853fddaf661c68f207346131a7e00d8ed2f 2023-08-26T13:30:53.271 DEBUG:teuthology.orchestra.run.smithi124:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2023-08-26T13:30:53.297 INFO:teuthology.orchestra.run.smithi124.stdout:18.0.0-5706.ge7eb9853.el8 2023-08-26T13:30:53.297 INFO:teuthology.packaging:The installed version of ceph is 18.0.0-5706.ge7eb9853.el8 2023-08-26T13:30:53.298 INFO:teuthology.task.install:The correct ceph version 18.0.0-5706.ge7eb9853 is installed. 2023-08-26T13:30:53.298 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2023-08-26T13:30:53.298 DEBUG:teuthology.orchestra.run.smithi031:> set -ex 2023-08-26T13:30:53.299 DEBUG:teuthology.orchestra.run.smithi031:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2023-08-26T13:30:53.326 DEBUG:teuthology.orchestra.run.smithi124:> set -ex 2023-08-26T13:30:53.327 DEBUG:teuthology.orchestra.run.smithi124:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2023-08-26T13:30:53.370 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2023-08-26T13:30:53.370 DEBUG:teuthology.orchestra.run.smithi031:> set -ex 2023-08-26T13:30:53.370 DEBUG:teuthology.orchestra.run.smithi031:> sudo dd of=/usr/bin/daemon-helper 2023-08-26T13:30:53.398 DEBUG:teuthology.orchestra.run.smithi031:> sudo chmod a=rx -- /usr/bin/daemon-helper 2023-08-26T13:30:53.465 DEBUG:teuthology.orchestra.run.smithi124:> set -ex 2023-08-26T13:30:53.466 DEBUG:teuthology.orchestra.run.smithi124:> sudo dd of=/usr/bin/daemon-helper 2023-08-26T13:30:53.497 DEBUG:teuthology.orchestra.run.smithi124:> sudo chmod a=rx -- /usr/bin/daemon-helper 2023-08-26T13:30:53.567 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2023-08-26T13:30:53.567 DEBUG:teuthology.orchestra.run.smithi031:> set -ex 2023-08-26T13:30:53.568 DEBUG:teuthology.orchestra.run.smithi031:> sudo dd of=/usr/bin/adjust-ulimits 2023-08-26T13:30:53.595 DEBUG:teuthology.orchestra.run.smithi031:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2023-08-26T13:30:53.662 DEBUG:teuthology.orchestra.run.smithi124:> set -ex 2023-08-26T13:30:53.662 DEBUG:teuthology.orchestra.run.smithi124:> sudo dd of=/usr/bin/adjust-ulimits 2023-08-26T13:30:53.691 DEBUG:teuthology.orchestra.run.smithi124:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2023-08-26T13:30:53.758 INFO:teuthology.task.install.util:Shipping 'stdin-killer'... 2023-08-26T13:30:53.759 DEBUG:teuthology.orchestra.run.smithi031:> set -ex 2023-08-26T13:30:53.759 DEBUG:teuthology.orchestra.run.smithi031:> sudo dd of=/usr/bin/stdin-killer 2023-08-26T13:30:53.786 DEBUG:teuthology.orchestra.run.smithi031:> sudo chmod a=rx -- /usr/bin/stdin-killer 2023-08-26T13:30:53.853 DEBUG:teuthology.orchestra.run.smithi124:> set -ex 2023-08-26T13:30:53.853 DEBUG:teuthology.orchestra.run.smithi124:> sudo dd of=/usr/bin/stdin-killer 2023-08-26T13:30:53.881 DEBUG:teuthology.orchestra.run.smithi124:> sudo chmod a=rx -- /usr/bin/stdin-killer 2023-08-26T13:30:53.949 INFO:teuthology.run_tasks:Running task ceph... 2023-08-26T13:30:54.020 INFO:tasks.ceph:Making ceph log dir writeable by non-root... 2023-08-26T13:30:54.021 DEBUG:teuthology.orchestra.run.smithi031:> sudo chmod 777 /var/log/ceph 2023-08-26T13:30:54.024 DEBUG:teuthology.orchestra.run.smithi124:> sudo chmod 777 /var/log/ceph 2023-08-26T13:30:54.051 INFO:tasks.ceph:Disabling ceph logrotate... 2023-08-26T13:30:54.051 DEBUG:teuthology.orchestra.run.smithi031:> sudo rm -f -- /etc/logrotate.d/ceph 2023-08-26T13:30:54.092 DEBUG:teuthology.orchestra.run.smithi124:> sudo rm -f -- /etc/logrotate.d/ceph 2023-08-26T13:30:54.118 INFO:tasks.ceph:Creating extra log directories... 2023-08-26T13:30:54.118 DEBUG:teuthology.orchestra.run.smithi031:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2023-08-26T13:30:54.158 DEBUG:teuthology.orchestra.run.smithi124:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2023-08-26T13:30:54.197 INFO:tasks.ceph:Creating ceph cluster cluster1... 2023-08-26T13:30:54.198 INFO:tasks.ceph:config {'conf': {'client': {'debug rbd': 20, 'debug rbd_mirror': 15, 'log to stderr': False, 'rbd default features': 125, 'rbd mirror image policy type': 'simple'}, 'client.mirror.0': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.1': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.2': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.3': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'global': {'mon client directed command retry': 5, 'ms inject socket failures': 5000}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20}, 'osd': {'bdev async discard': True, 'bdev enable discard': True, 'bluestore allocator': 'stupid', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore'}}, 'fs': 'xfs', 'mkfs_options': None, 'mount_options': None, 'skip_mgr_daemons': False, 'log_ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(OSD_SLOW_PING_TIME'], 'cpu_profile': set(), 'cluster': 'cluster1', 'mon_bind_msgr2': True, 'mon_bind_addrvec': True} 2023-08-26T13:30:54.198 INFO:tasks.ceph:ctx.config {'archive_path': '/home/teuthworker/archive/yuriw-2023-08-25_17:40:36-rbd-main-distro-default-smithi/7380431', 'branch': 'main', 'description': 'rbd/mirror-thrash/{base/install clients/mirror cluster/{2-node openstack} msgr-failures/few objectstore/bluestore-stupid policy/simple rbd-mirror/four-per-cluster supported-random-distro$/{rhel_8} workloads/rbd-mirror-snapshot-stress-workunit-exclusive-lock}', 'email': 'yweinste@redhat.com', 'first_in_suite': False, 'job_id': '7380431', 'kernel': {'kdb': True, 'sha1': 'distro'}, 'ktype': 'distro', 'last_in_suite': False, 'machine_type': 'smithi', 'meta': [{'desc': 'run two ceph clusters and install rbd-mirror'}, {'desc': 'configure the permissions for client.mirror'}, {'desc': '2 ceph clusters with 1 mon and 3 osds each'}, {'desc': 'run four rbd-mirror daemons per cluster'}, {'desc': 'run the rbd_mirror_stress.sh workunit to test the rbd-mirror daemon'}], 'name': 'yuriw-2023-08-25_17:40:36-rbd-main-distro-default-smithi', 'no_nested_subset': False, 'nuke-on-error': True, 'openstack': [{'volumes': {'count': 3, 'size': 30}}], 'os_type': 'rhel', 'os_version': '8.6', 'overrides': {'admin_socket': {'branch': 'main'}, 'ceph': {'conf': {'client': {'debug rbd': 20, 'debug rbd_mirror': 15, 'log to stderr': False, 'rbd default features': 125, 'rbd mirror image policy type': 'simple'}, 'client.mirror.0': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.1': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.2': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.3': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'global': {'mon client directed command retry': 5, 'ms inject socket failures': 5000}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20}, 'osd': {'bdev async discard': True, 'bdev enable discard': True, 'bluestore allocator': 'stupid', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore'}}, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(OSD_SLOW_PING_TIME'], 'sha1': 'e7eb9853fddaf661c68f207346131a7e00d8ed2f'}, 'ceph-deploy': {'bluestore': True, 'conf': {'client': {'log file': '/var/log/ceph/ceph-$name.$pid.log'}, 'mon': {}, 'osd': {'bdev async discard': True, 'bdev enable discard': True, 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore'}}, 'fs': 'xfs'}, 'install': {'ceph': {'flavor': 'default', 'sha1': 'e7eb9853fddaf661c68f207346131a7e00d8ed2f'}}, 'selinux': {'whitelist': ['scontext=system_u:system_r:logrotate_t:s0']}, 'thrashosds': {'bdev_inject_crash': 2, 'bdev_inject_crash_probability': 0.5}, 'workunit': {'branch': 'main', 'sha1': 'e7eb9853fddaf661c68f207346131a7e00d8ed2f'}}, 'owner': 'scheduled_yuriw@teuthology', 'priority': 75, 'repo': 'https://github.com/ceph/ceph-ci.git', 'roles': [['cluster1.mon.a', 'cluster1.mgr.x', 'cluster2.mgr.x', 'cluster1.osd.0', 'cluster1.osd.1', 'cluster1.osd.2', 'cluster1.client.0', 'cluster2.client.0'], ['cluster2.mon.a', 'cluster2.osd.0', 'cluster2.osd.1', 'cluster2.osd.2', 'cluster1.client.mirror', 'cluster1.client.mirror.0', 'cluster1.client.mirror.1', 'cluster1.client.mirror.2', 'cluster1.client.mirror.3', 'cluster1.client.mirror.4', 'cluster1.client.mirror.5', 'cluster1.client.mirror.6', 'cluster2.client.mirror', 'cluster2.client.mirror.0', 'cluster2.client.mirror.1', 'cluster2.client.mirror.2', 'cluster2.client.mirror.3', 'cluster2.client.mirror.4', 'cluster2.client.mirror.5', 'cluster2.client.mirror.6']], 'seed': 5715, 'sha1': 'e7eb9853fddaf661c68f207346131a7e00d8ed2f', 'sleep_before_teardown': 0, 'subset': '111/120000', 'suite': 'rbd', 'suite_branch': 'main', 'suite_path': '/home/teuthworker/src/github.com_ceph_ceph-c_e7eb9853fddaf661c68f207346131a7e00d8ed2f/qa', 'suite_relpath': 'qa', 'suite_repo': 'https://github.com/ceph/ceph-ci.git', 'suite_sha1': 'e7eb9853fddaf661c68f207346131a7e00d8ed2f', 'targets': {'smithi031.front.sepia.ceph.com': 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCpUxo5ay/ucdkX9rleNMm8Ofp24XRXUtJPOK3Ji3XBpTiUgmmwneaY6316+3CG+k4K0p3IebanuyTvZApjm02/dNCvu8yD3rjEpWVgIuqYH8O3lEdN9DXcCX+7xHJE0m18+ZPsJQVsNrNJxY+5zqyNnBc8jpZq2LStsWwPOKPWhauANRkOBFSGDoLppWZbLx/yJs+R0qoNLLK9LpKQTUnqsapXvbD27OpPahzj/qbNxFPPyY6XrrldZ6qEO+EjJOujFLH3gzv0kJu10DCuzxrpz6ston4RPi1O3GgcHnF4Mqr/W9VwZ94UTMh+k4dIFRVqJEKrcwRHa5kFsvH/WhVx0Ygr/MiV/Q86mmV7JT+/MpzIRTtWyei5L9IBeZIE9cwzj+mOk6SD/t1roIVEGVV/uF2AyFDVBPr6F8oZ822OGpUyyaDsWTitPNdTX4TC3zdSrLc9tyDa1P74eH/ZLeTSf+C6pM/cepOhQaa6XzpDiPSuJUm9HOQxUlEAvcqa7Ac=', 'smithi124.front.sepia.ceph.com': 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCpUxo5ay/ucdkX9rleNMm8Ofp24XRXUtJPOK3Ji3XBpTiUgmmwneaY6316+3CG+k4K0p3IebanuyTvZApjm02/dNCvu8yD3rjEpWVgIuqYH8O3lEdN9DXcCX+7xHJE0m18+ZPsJQVsNrNJxY+5zqyNnBc8jpZq2LStsWwPOKPWhauANRkOBFSGDoLppWZbLx/yJs+R0qoNLLK9LpKQTUnqsapXvbD27OpPahzj/qbNxFPPyY6XrrldZ6qEO+EjJOujFLH3gzv0kJu10DCuzxrpz6ston4RPi1O3GgcHnF4Mqr/W9VwZ94UTMh+k4dIFRVqJEKrcwRHa5kFsvH/WhVx0Ygr/MiV/Q86mmV7JT+/MpzIRTtWyei5L9IBeZIE9cwzj+mOk6SD/t1roIVEGVV/uF2AyFDVBPr6F8oZ822OGpUyyaDsWTitPNdTX4TC3zdSrLc9tyDa1P74eH/ZLeTSf+C6pM/cepOhQaa6XzpDiPSuJUm9HOQxUlEAvcqa7Ac='}, 'tasks': [{'internal.check_packages': None}, {'internal.buildpackages_prep': None}, {'internal.save_config': None}, {'internal.check_lock': None}, {'internal.add_remotes': None}, {'console_log': None}, {'internal.connect': None}, {'internal.push_inventory': None}, {'internal.serialize_remote_roles': None}, {'internal.check_conflict': None}, {'internal.check_ceph_data': None}, {'internal.vm_setup': None}, {'kernel': {'kdb': True, 'sha1': 'distro'}}, {'internal.base': None}, {'internal.archive_upload': None}, {'internal.archive': None}, {'internal.coredump': None}, {'internal.sudo': None}, {'internal.syslog': None}, {'internal.timer': None}, {'pcp': None}, {'selinux': None}, {'ansible.cephlab': None}, {'clock': None}, {'install': {'extra_packages': ['rbd-mirror'], 'flavor': 'default', 'sha1': 'e7eb9853fddaf661c68f207346131a7e00d8ed2f'}}, {'ceph': {'cluster': 'cluster1', 'conf': {'client': {'debug rbd': 20, 'debug rbd_mirror': 15, 'log to stderr': False, 'rbd default features': 125, 'rbd mirror image policy type': 'simple'}, 'client.mirror.0': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.1': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.2': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.3': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'global': {'mon client directed command retry': 5, 'ms inject socket failures': 5000}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20}, 'osd': {'bdev async discard': True, 'bdev enable discard': True, 'bluestore allocator': 'stupid', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore'}}, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(OSD_SLOW_PING_TIME'], 'sha1': 'e7eb9853fddaf661c68f207346131a7e00d8ed2f'}}, {'ceph': {'cluster': 'cluster2'}}, {'exec': {'cluster1.client.mirror.0': ["sudo ceph --cluster cluster1 auth caps client.mirror mon 'profile rbd-mirror-peer' osd 'profile rbd'", "sudo ceph --cluster cluster1 auth caps client.mirror.0 mon 'profile rbd-mirror' osd 'profile rbd'", "sudo ceph --cluster cluster1 auth caps client.mirror.1 mon 'profile rbd-mirror' osd 'profile rbd'", "sudo ceph --cluster cluster1 auth caps client.mirror.2 mon 'profile rbd-mirror' osd 'profile rbd'", "sudo ceph --cluster cluster1 auth caps client.mirror.3 mon 'profile rbd-mirror' osd 'profile rbd'", "sudo ceph --cluster cluster2 auth caps client.mirror mon 'profile rbd-mirror-peer' osd 'profile rbd'", "sudo ceph --cluster cluster2 auth caps client.mirror.0 mon 'profile rbd-mirror' osd 'profile rbd'", "sudo ceph --cluster cluster2 auth caps client.mirror.1 mon 'profile rbd-mirror' osd 'profile rbd'", "sudo ceph --cluster cluster2 auth caps client.mirror.2 mon 'profile rbd-mirror' osd 'profile rbd'", "sudo ceph --cluster cluster2 auth caps client.mirror.3 mon 'profile rbd-mirror' osd 'profile rbd'"]}}, {'rbd-mirror': {'client': 'cluster1.client.mirror.0', 'thrash': True}}, {'rbd-mirror': {'client': 'cluster1.client.mirror.1', 'thrash': True}}, {'rbd-mirror': {'client': 'cluster1.client.mirror.2', 'thrash': True}}, {'rbd-mirror': {'client': 'cluster1.client.mirror.3', 'thrash': True}}, {'rbd-mirror': {'client': 'cluster2.client.mirror.0', 'thrash': True}}, {'rbd-mirror': {'client': 'cluster2.client.mirror.1', 'thrash': True}}, {'rbd-mirror': {'client': 'cluster2.client.mirror.2', 'thrash': True}}, {'rbd-mirror': {'client': 'cluster2.client.mirror.3', 'thrash': True}}, {'rbd-mirror-thrash': {'cluster': 'cluster1'}}, {'rbd-mirror-thrash': {'cluster': 'cluster2'}}, {'workunit': {'clients': {'cluster1.client.mirror': ['rbd/rbd_mirror_stress.sh']}, 'env': {'CEPH_ARGS': '', 'MIRROR_IMAGE_MODE': 'snapshot', 'MIRROR_POOL_MODE': 'image', 'RBD_IMAGE_FEATURES': 'layering,exclusive-lock', 'RBD_MIRROR_INSTANCES': '4', 'RBD_MIRROR_USE_EXISTING_CLUSTER': '1', 'RBD_MIRROR_USE_RBD_MIRROR': '1'}, 'timeout': '6h'}}], 'teuthology': {'fragments_dropped': [], 'meta': {}, 'postmerge': []}, 'teuthology_branch': 'main', 'teuthology_sha1': '449a1bc2027504e7b3c3d7b30fa4178906581da7', 'timestamp': '2023-08-25_17:40:36', 'tube': 'smithi', 'user': 'yuriw', 'verbose': True, 'worker_log': '/home/teuthworker/archive/worker_logs/dispatcher.smithi.2002'} 2023-08-26T13:30:54.199 DEBUG:teuthology.orchestra.run.smithi031:> install -d -m0755 -- /home/ubuntu/cephtest/cluster1.data 2023-08-26T13:30:54.225 DEBUG:teuthology.orchestra.run.smithi124:> install -d -m0755 -- /home/ubuntu/cephtest/cluster1.data 2023-08-26T13:30:54.242 DEBUG:teuthology.orchestra.run.smithi031:> sudo install -d -m0777 -- /var/run/ceph 2023-08-26T13:30:54.282 DEBUG:teuthology.orchestra.run.smithi124:> sudo install -d -m0777 -- /var/run/ceph 2023-08-26T13:30:54.309 DEBUG:teuthology.orchestra.run.smithi031:> set -ex 2023-08-26T13:30:54.309 DEBUG:teuthology.orchestra.run.smithi031:> dd if=/scratch_devs of=/dev/stdout 2023-08-26T13:30:54.364 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2023-08-26T13:30:54.364 DEBUG:teuthology.orchestra.run.smithi031:> stat /dev/vg_nvme/lv_1 2023-08-26T13:30:54.421 INFO:teuthology.orchestra.run.smithi031.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2023-08-26T13:30:54.421 INFO:teuthology.orchestra.run.smithi031.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-08-26T13:30:54.421 INFO:teuthology.orchestra.run.smithi031.stdout:Device: 6h/6d Inode: 384704 Links: 1 2023-08-26T13:30:54.421 INFO:teuthology.orchestra.run.smithi031.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-08-26T13:30:54.422 INFO:teuthology.orchestra.run.smithi031.stdout:Context: system_u:object_r:device_t:s0 2023-08-26T13:30:54.422 INFO:teuthology.orchestra.run.smithi031.stdout:Access: 2023-08-26 13:30:36.407379714 +0000 2023-08-26T13:30:54.422 INFO:teuthology.orchestra.run.smithi031.stdout:Modify: 2023-08-26 13:24:54.089386561 +0000 2023-08-26T13:30:54.422 INFO:teuthology.orchestra.run.smithi031.stdout:Change: 2023-08-26 13:24:54.089386561 +0000 2023-08-26T13:30:54.422 INFO:teuthology.orchestra.run.smithi031.stdout: Birth: - 2023-08-26T13:30:54.423 DEBUG:teuthology.orchestra.run.smithi031:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2023-08-26T13:30:54.486 INFO:teuthology.orchestra.run.smithi031.stderr:1+0 records in 2023-08-26T13:30:54.486 INFO:teuthology.orchestra.run.smithi031.stderr:1+0 records out 2023-08-26T13:30:54.487 INFO:teuthology.orchestra.run.smithi031.stderr:512 bytes copied, 0.000250896 s, 2.0 MB/s 2023-08-26T13:30:54.488 DEBUG:teuthology.orchestra.run.smithi031:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2023-08-26T13:30:54.546 DEBUG:teuthology.orchestra.run.smithi031:> stat /dev/vg_nvme/lv_2 2023-08-26T13:30:54.603 INFO:teuthology.orchestra.run.smithi031.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2023-08-26T13:30:54.604 INFO:teuthology.orchestra.run.smithi031.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-08-26T13:30:54.604 INFO:teuthology.orchestra.run.smithi031.stdout:Device: 6h/6d Inode: 389780 Links: 1 2023-08-26T13:30:54.604 INFO:teuthology.orchestra.run.smithi031.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-08-26T13:30:54.604 INFO:teuthology.orchestra.run.smithi031.stdout:Context: system_u:object_r:device_t:s0 2023-08-26T13:30:54.605 INFO:teuthology.orchestra.run.smithi031.stdout:Access: 2023-08-26 13:30:36.407379714 +0000 2023-08-26T13:30:54.605 INFO:teuthology.orchestra.run.smithi031.stdout:Modify: 2023-08-26 13:24:54.593378533 +0000 2023-08-26T13:30:54.605 INFO:teuthology.orchestra.run.smithi031.stdout:Change: 2023-08-26 13:24:54.593378533 +0000 2023-08-26T13:30:54.605 INFO:teuthology.orchestra.run.smithi031.stdout: Birth: - 2023-08-26T13:30:54.605 DEBUG:teuthology.orchestra.run.smithi031:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2023-08-26T13:30:54.670 INFO:teuthology.orchestra.run.smithi031.stderr:1+0 records in 2023-08-26T13:30:54.670 INFO:teuthology.orchestra.run.smithi031.stderr:1+0 records out 2023-08-26T13:30:54.671 INFO:teuthology.orchestra.run.smithi031.stderr:512 bytes copied, 0.000321378 s, 1.6 MB/s 2023-08-26T13:30:54.672 DEBUG:teuthology.orchestra.run.smithi031:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2023-08-26T13:30:54.730 DEBUG:teuthology.orchestra.run.smithi031:> stat /dev/vg_nvme/lv_3 2023-08-26T13:30:54.787 INFO:teuthology.orchestra.run.smithi031.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2023-08-26T13:30:54.788 INFO:teuthology.orchestra.run.smithi031.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-08-26T13:30:54.788 INFO:teuthology.orchestra.run.smithi031.stdout:Device: 6h/6d Inode: 393544 Links: 1 2023-08-26T13:30:54.788 INFO:teuthology.orchestra.run.smithi031.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-08-26T13:30:54.788 INFO:teuthology.orchestra.run.smithi031.stdout:Context: system_u:object_r:device_t:s0 2023-08-26T13:30:54.788 INFO:teuthology.orchestra.run.smithi031.stdout:Access: 2023-08-26 13:30:36.407379714 +0000 2023-08-26T13:30:54.788 INFO:teuthology.orchestra.run.smithi031.stdout:Modify: 2023-08-26 13:24:55.103370410 +0000 2023-08-26T13:30:54.789 INFO:teuthology.orchestra.run.smithi031.stdout:Change: 2023-08-26 13:24:55.103370410 +0000 2023-08-26T13:30:54.789 INFO:teuthology.orchestra.run.smithi031.stdout: Birth: - 2023-08-26T13:30:54.789 DEBUG:teuthology.orchestra.run.smithi031:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2023-08-26T13:30:54.853 INFO:teuthology.orchestra.run.smithi031.stderr:1+0 records in 2023-08-26T13:30:54.853 INFO:teuthology.orchestra.run.smithi031.stderr:1+0 records out 2023-08-26T13:30:54.853 INFO:teuthology.orchestra.run.smithi031.stderr:512 bytes copied, 0.000214646 s, 2.4 MB/s 2023-08-26T13:30:54.854 DEBUG:teuthology.orchestra.run.smithi031:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2023-08-26T13:30:54.912 DEBUG:teuthology.orchestra.run.smithi031:> stat /dev/vg_nvme/lv_4 2023-08-26T13:30:54.969 INFO:teuthology.orchestra.run.smithi031.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2023-08-26T13:30:54.970 INFO:teuthology.orchestra.run.smithi031.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-08-26T13:30:54.970 INFO:teuthology.orchestra.run.smithi031.stdout:Device: 6h/6d Inode: 394664 Links: 1 2023-08-26T13:30:54.970 INFO:teuthology.orchestra.run.smithi031.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-08-26T13:30:54.970 INFO:teuthology.orchestra.run.smithi031.stdout:Context: system_u:object_r:device_t:s0 2023-08-26T13:30:54.970 INFO:teuthology.orchestra.run.smithi031.stdout:Access: 2023-08-26 13:30:36.407379714 +0000 2023-08-26T13:30:54.971 INFO:teuthology.orchestra.run.smithi031.stdout:Modify: 2023-08-26 13:24:55.626362079 +0000 2023-08-26T13:30:54.971 INFO:teuthology.orchestra.run.smithi031.stdout:Change: 2023-08-26 13:24:55.626362079 +0000 2023-08-26T13:30:54.971 INFO:teuthology.orchestra.run.smithi031.stdout: Birth: - 2023-08-26T13:30:54.971 DEBUG:teuthology.orchestra.run.smithi031:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2023-08-26T13:30:55.035 INFO:teuthology.orchestra.run.smithi031.stderr:1+0 records in 2023-08-26T13:30:55.036 INFO:teuthology.orchestra.run.smithi031.stderr:1+0 records out 2023-08-26T13:30:55.036 INFO:teuthology.orchestra.run.smithi031.stderr:512 bytes copied, 0.000201129 s, 2.5 MB/s 2023-08-26T13:30:55.037 DEBUG:teuthology.orchestra.run.smithi031:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2023-08-26T13:30:55.095 INFO:tasks.ceph:osd dev map: {'cluster1.osd.0': '/dev/vg_nvme/lv_1', 'cluster1.osd.1': '/dev/vg_nvme/lv_2', 'cluster1.osd.2': '/dev/vg_nvme/lv_3'} 2023-08-26T13:30:55.095 INFO:tasks.ceph:remote_to_roles_to_devs: {Remote(name='ubuntu@smithi031.front.sepia.ceph.com'): {'cluster1.osd.0': '/dev/vg_nvme/lv_1', 'cluster1.osd.1': '/dev/vg_nvme/lv_2', 'cluster1.osd.2': '/dev/vg_nvme/lv_3'}} 2023-08-26T13:30:55.095 INFO:tasks.ceph:Generating config... 2023-08-26T13:30:55.097 INFO:tasks.ceph:[client] debug rbd = 20 2023-08-26T13:30:55.097 INFO:tasks.ceph:[client] debug rbd_mirror = 15 2023-08-26T13:30:55.097 INFO:tasks.ceph:[client] log to stderr = False 2023-08-26T13:30:55.097 INFO:tasks.ceph:[client] rbd default features = 125 2023-08-26T13:30:55.098 INFO:tasks.ceph:[client] rbd mirror image policy type = simple 2023-08-26T13:30:55.098 INFO:tasks.ceph:[client.mirror.0] admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2023-08-26T13:30:55.098 INFO:tasks.ceph:[client.mirror.0] pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2023-08-26T13:30:55.098 INFO:tasks.ceph:[client.mirror.1] admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2023-08-26T13:30:55.099 INFO:tasks.ceph:[client.mirror.1] pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2023-08-26T13:30:55.099 INFO:tasks.ceph:[client.mirror.2] admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2023-08-26T13:30:55.099 INFO:tasks.ceph:[client.mirror.2] pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2023-08-26T13:30:55.099 INFO:tasks.ceph:[client.mirror.3] admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2023-08-26T13:30:55.100 INFO:tasks.ceph:[client.mirror.3] pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2023-08-26T13:30:55.100 INFO:tasks.ceph:[global] mon client directed command retry = 5 2023-08-26T13:30:55.100 INFO:tasks.ceph:[global] ms inject socket failures = 5000 2023-08-26T13:30:55.100 INFO:tasks.ceph:[mgr] debug mgr = 20 2023-08-26T13:30:55.101 INFO:tasks.ceph:[mgr] debug ms = 1 2023-08-26T13:30:55.101 INFO:tasks.ceph:[mon] debug mon = 20 2023-08-26T13:30:55.101 INFO:tasks.ceph:[mon] debug ms = 1 2023-08-26T13:30:55.101 INFO:tasks.ceph:[mon] debug paxos = 20 2023-08-26T13:30:55.102 INFO:tasks.ceph:[osd] bdev async discard = True 2023-08-26T13:30:55.102 INFO:tasks.ceph:[osd] bdev enable discard = True 2023-08-26T13:30:55.102 INFO:tasks.ceph:[osd] bluestore allocator = stupid 2023-08-26T13:30:55.102 INFO:tasks.ceph:[osd] bluestore block size = 96636764160 2023-08-26T13:30:55.103 INFO:tasks.ceph:[osd] bluestore fsck on mount = True 2023-08-26T13:30:55.103 INFO:tasks.ceph:[osd] debug bluefs = 1/20 2023-08-26T13:30:55.103 INFO:tasks.ceph:[osd] debug bluestore = 1/20 2023-08-26T13:30:55.103 INFO:tasks.ceph:[osd] debug ms = 1 2023-08-26T13:30:55.104 INFO:tasks.ceph:[osd] debug osd = 20 2023-08-26T13:30:55.104 INFO:tasks.ceph:[osd] debug rocksdb = 4/10 2023-08-26T13:30:55.104 INFO:tasks.ceph:[osd] mon osd backfillfull_ratio = 0.85 2023-08-26T13:30:55.104 INFO:tasks.ceph:[osd] mon osd full ratio = 0.9 2023-08-26T13:30:55.105 INFO:tasks.ceph:[osd] mon osd nearfull ratio = 0.8 2023-08-26T13:30:55.105 INFO:tasks.ceph:[osd] osd failsafe full ratio = 0.95 2023-08-26T13:30:55.105 INFO:tasks.ceph:[osd] osd objectstore = bluestore 2023-08-26T13:30:55.105 INFO:tasks.ceph:Setting up cluster1.mon.a... 2023-08-26T13:30:55.106 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring /etc/ceph/cluster1.keyring 2023-08-26T13:30:55.191 INFO:teuthology.orchestra.run.smithi031.stdout:creating /etc/ceph/cluster1.keyring 2023-08-26T13:30:55.193 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=mon. /etc/ceph/cluster1.keyring 2023-08-26T13:30:55.239 DEBUG:teuthology.orchestra.run.smithi031:> sudo chmod 0644 /etc/ceph/cluster1.keyring 2023-08-26T13:30:55.266 DEBUG:tasks.ceph:Ceph mon addresses: [('cluster1.mon.a', '172.21.15.31')] 2023-08-26T13:30:55.267 DEBUG:tasks.ceph:writing out conf {'global': {'chdir': '', 'pid file': '/var/run/ceph/$cluster-$name.pid', 'auth supported': 'cephx', 'filestore xattr use omap': 'true', 'mon clock drift allowed': '1.000', 'osd crush chooseleaf type': '0', 'auth debug': 'true', 'ms die on old message': 'true', 'ms die on bug': 'true', 'mon max pg per osd': '10000', 'mon pg warn max object skew': '0', 'osd_pool_default_pg_autoscale_mode': 'off', 'osd pool default size': '2', 'mon osd allow primary affinity': 'true', 'mon osd allow pg remap': 'true', 'mon warn on legacy crush tunables': 'false', 'mon warn on crush straw calc version zero': 'false', 'mon warn on no sortbitwise': 'false', 'mon warn on osd down out interval zero': 'false', 'mon warn on too few osds': 'false', 'mon_warn_on_pool_pg_num_not_power_of_two': 'false', 'mon_warn_on_pool_no_redundancy': 'false', 'mon_allow_pool_size_one': 'true', 'osd pool default erasure code profile': 'plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd', 'osd default data pool replay window': '5', 'mon allow pool delete': 'true', 'mon cluster log file level': 'debug', 'debug asserts on shutdown': 'true', 'mon health detail to clog': 'false', 'mon host': '172.21.15.31', 'mon client directed command retry': 5, 'ms inject socket failures': 5000}, 'osd': {'osd journal size': '100', 'osd scrub load threshold': '5.0', 'osd scrub max interval': '600', 'osd mclock profile': 'high_recovery_ops', 'osd recover clone overlap': 'true', 'osd recovery max chunk': '1048576', 'osd debug shutdown': 'true', 'osd debug op order': 'true', 'osd debug verify stray on activate': 'true', 'osd open classes on start': 'true', 'osd debug pg log writeout': 'true', 'osd deep scrub update digest min age': '30', 'osd map max advance': '10', 'journal zero on create': 'true', 'filestore ondisk finisher threads': '3', 'filestore apply finisher threads': '3', 'bdev debug aio': 'true', 'osd debug misdirected ops': 'true', 'bdev async discard': True, 'bdev enable discard': True, 'bluestore allocator': 'stupid', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore'}, 'mgr': {'debug ms': 1, 'debug mgr': 20, 'debug mon': '20', 'debug auth': '20', 'mon reweight min pgs per osd': '4', 'mon reweight min bytes per osd': '10', 'mgr/telemetry/nag': 'false'}, 'mon': {'debug ms': 1, 'debug mon': 20, 'debug paxos': 20, 'debug auth': '20', 'mon data avail warn': '5', 'mon mgr mkfs grace': '240', 'mon reweight min pgs per osd': '4', 'mon osd reporter subtree level': 'osd', 'mon osd prime pg temp': 'true', 'mon reweight min bytes per osd': '10', 'auth mon ticket ttl': '660', 'auth service ticket ttl': '240', 'mon_warn_on_insecure_global_id_reclaim': 'false', 'mon_warn_on_insecure_global_id_reclaim_allowed': 'false', 'mon_down_mkfs_grace': '2m', 'mon_warn_on_filestore_osds': 'false'}, 'client': {'rgw cache enabled': 'true', 'rgw enable ops log': 'true', 'rgw enable usage log': 'true', 'log file': '/var/log/ceph/$cluster-$name.$pid.log', 'admin socket': '/var/run/ceph/$cluster-$name.$pid.asok', 'debug rbd': 20, 'debug rbd_mirror': 15, 'log to stderr': False, 'rbd default features': 125, 'rbd mirror image policy type': 'simple'}, 'mon.a': {}, 'client.mirror.0': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.1': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.2': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.3': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}} 2023-08-26T13:30:55.268 DEBUG:teuthology.orchestra.run.smithi031:> set -ex 2023-08-26T13:30:55.268 DEBUG:teuthology.orchestra.run.smithi031:> dd of=/home/ubuntu/cephtest/ceph.tmp.conf 2023-08-26T13:30:55.323 DEBUG:teuthology.orchestra.run.smithi031:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage monmaptool -c /home/ubuntu/cephtest/ceph.tmp.conf --create --clobber --enable-all-features --add a 172.21.15.31 --print /home/ubuntu/cephtest/cluster1.monmap 2023-08-26T13:30:55.405 INFO:teuthology.orchestra.run.smithi031.stdout:monmaptool: monmap file /home/ubuntu/cephtest/cluster1.monmap 2023-08-26T13:30:55.406 INFO:teuthology.orchestra.run.smithi031.stdout:monmaptool: generated fsid f17edf88-b4bf-49db-bf48-f12793acdc0b 2023-08-26T13:30:55.406 INFO:teuthology.orchestra.run.smithi031.stdout:setting min_mon_release = pacific 2023-08-26T13:30:55.406 INFO:teuthology.orchestra.run.smithi031.stdout:epoch 0 2023-08-26T13:30:55.406 INFO:teuthology.orchestra.run.smithi031.stdout:fsid f17edf88-b4bf-49db-bf48-f12793acdc0b 2023-08-26T13:30:55.406 INFO:teuthology.orchestra.run.smithi031.stdout:last_changed 2023-08-26T13:30:55.405414+0000 2023-08-26T13:30:55.407 INFO:teuthology.orchestra.run.smithi031.stdout:created 2023-08-26T13:30:55.405414+0000 2023-08-26T13:30:55.407 INFO:teuthology.orchestra.run.smithi031.stdout:min_mon_release 16 (pacific) 2023-08-26T13:30:55.407 INFO:teuthology.orchestra.run.smithi031.stdout:election_strategy: 1 2023-08-26T13:30:55.407 INFO:teuthology.orchestra.run.smithi031.stdout:0: [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] mon.a 2023-08-26T13:30:55.407 INFO:teuthology.orchestra.run.smithi031.stdout:monmaptool: writing epoch 0 to /home/ubuntu/cephtest/cluster1.monmap (1 monitors) 2023-08-26T13:30:55.409 DEBUG:teuthology.orchestra.run.smithi031:> rm -- /home/ubuntu/cephtest/ceph.tmp.conf 2023-08-26T13:30:55.465 INFO:tasks.ceph:Writing /etc/ceph/cluster1.conf for FSID f17edf88-b4bf-49db-bf48-f12793acdc0b... 2023-08-26T13:30:55.466 DEBUG:teuthology.orchestra.run.smithi031:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/cluster1.conf && sudo chmod 0644 /etc/ceph/cluster1.conf > /dev/null 2023-08-26T13:30:55.507 DEBUG:teuthology.orchestra.run.smithi124:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/cluster1.conf && sudo chmod 0644 /etc/ceph/cluster1.conf > /dev/null 2023-08-26T13:30:55.551 INFO:teuthology.orchestra.run.smithi031.stdout:[global] 2023-08-26T13:30:55.552 INFO:teuthology.orchestra.run.smithi031.stdout: chdir = "" 2023-08-26T13:30:55.552 INFO:teuthology.orchestra.run.smithi031.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2023-08-26T13:30:55.552 INFO:teuthology.orchestra.run.smithi031.stdout: auth supported = cephx 2023-08-26T13:30:55.552 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-08-26T13:30:55.552 INFO:teuthology.orchestra.run.smithi031.stdout: filestore xattr use omap = true 2023-08-26T13:30:55.553 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-08-26T13:30:55.553 INFO:teuthology.orchestra.run.smithi031.stdout: mon clock drift allowed = 1.000 2023-08-26T13:30:55.553 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-08-26T13:30:55.553 INFO:teuthology.orchestra.run.smithi031.stdout: osd crush chooseleaf type = 0 2023-08-26T13:30:55.553 INFO:teuthology.orchestra.run.smithi031.stdout: auth debug = true 2023-08-26T13:30:55.553 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-08-26T13:30:55.554 INFO:teuthology.orchestra.run.smithi031.stdout: ms die on old message = true 2023-08-26T13:30:55.554 INFO:teuthology.orchestra.run.smithi031.stdout: ms die on bug = true 2023-08-26T13:30:55.554 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-08-26T13:30:55.554 INFO:teuthology.orchestra.run.smithi031.stdout: mon max pg per osd = 10000 # >= luminous 2023-08-26T13:30:55.554 INFO:teuthology.orchestra.run.smithi031.stdout: mon pg warn max object skew = 0 2023-08-26T13:30:55.555 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-08-26T13:30:55.555 INFO:teuthology.orchestra.run.smithi031.stdout: # disable pg_autoscaler by default for new pools 2023-08-26T13:30:55.555 INFO:teuthology.orchestra.run.smithi031.stdout: osd_pool_default_pg_autoscale_mode = off 2023-08-26T13:30:55.555 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-08-26T13:30:55.555 INFO:teuthology.orchestra.run.smithi031.stdout: osd pool default size = 2 2023-08-26T13:30:55.556 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-08-26T13:30:55.556 INFO:teuthology.orchestra.run.smithi031.stdout: mon osd allow primary affinity = true 2023-08-26T13:30:55.556 INFO:teuthology.orchestra.run.smithi031.stdout: mon osd allow pg remap = true 2023-08-26T13:30:55.556 INFO:teuthology.orchestra.run.smithi031.stdout: mon warn on legacy crush tunables = false 2023-08-26T13:30:55.556 INFO:teuthology.orchestra.run.smithi031.stdout: mon warn on crush straw calc version zero = false 2023-08-26T13:30:55.556 INFO:teuthology.orchestra.run.smithi031.stdout: mon warn on no sortbitwise = false 2023-08-26T13:30:55.557 INFO:teuthology.orchestra.run.smithi031.stdout: mon warn on osd down out interval zero = false 2023-08-26T13:30:55.557 INFO:teuthology.orchestra.run.smithi031.stdout: mon warn on too few osds = false 2023-08-26T13:30:55.557 INFO:teuthology.orchestra.run.smithi031.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2023-08-26T13:30:55.557 INFO:teuthology.orchestra.run.smithi031.stdout: mon_warn_on_pool_no_redundancy = false 2023-08-26T13:30:55.557 INFO:teuthology.orchestra.run.smithi031.stdout: mon_allow_pool_size_one = true 2023-08-26T13:30:55.558 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-08-26T13:30:55.558 INFO:teuthology.orchestra.run.smithi031.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2023-08-26T13:30:55.558 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-08-26T13:30:55.558 INFO:teuthology.orchestra.run.smithi031.stdout: osd default data pool replay window = 5 2023-08-26T13:30:55.558 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-08-26T13:30:55.559 INFO:teuthology.orchestra.run.smithi031.stdout: mon allow pool delete = true 2023-08-26T13:30:55.559 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-08-26T13:30:55.559 INFO:teuthology.orchestra.run.smithi031.stdout: mon cluster log file level = debug 2023-08-26T13:30:55.559 INFO:teuthology.orchestra.run.smithi031.stdout: debug asserts on shutdown = true 2023-08-26T13:30:55.559 INFO:teuthology.orchestra.run.smithi031.stdout: mon health detail to clog = false 2023-08-26T13:30:55.559 INFO:teuthology.orchestra.run.smithi031.stdout: mon host = 172.21.15.31 2023-08-26T13:30:55.560 INFO:teuthology.orchestra.run.smithi031.stdout: mon client directed command retry = 5 2023-08-26T13:30:55.560 INFO:teuthology.orchestra.run.smithi031.stdout: ms inject socket failures = 5000 2023-08-26T13:30:55.560 INFO:teuthology.orchestra.run.smithi031.stdout: fsid = f17edf88-b4bf-49db-bf48-f12793acdc0b 2023-08-26T13:30:55.560 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-08-26T13:30:55.560 INFO:teuthology.orchestra.run.smithi031.stdout:[osd] 2023-08-26T13:30:55.561 INFO:teuthology.orchestra.run.smithi031.stdout: osd journal size = 100 2023-08-26T13:30:55.561 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-08-26T13:30:55.561 INFO:teuthology.orchestra.run.smithi031.stdout: osd scrub load threshold = 5.0 2023-08-26T13:30:55.561 INFO:teuthology.orchestra.run.smithi031.stdout: osd scrub max interval = 600 2023-08-26T13:30:55.561 INFO:teuthology.orchestra.run.smithi031.stdout: osd mclock profile = high_recovery_ops 2023-08-26T13:30:55.561 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-08-26T13:30:55.562 INFO:teuthology.orchestra.run.smithi031.stdout: osd recover clone overlap = true 2023-08-26T13:30:55.562 INFO:teuthology.orchestra.run.smithi031.stdout: osd recovery max chunk = 1048576 2023-08-26T13:30:55.562 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-08-26T13:30:55.562 INFO:teuthology.orchestra.run.smithi031.stdout: osd debug shutdown = true 2023-08-26T13:30:55.562 INFO:teuthology.orchestra.run.smithi031.stdout: osd debug op order = true 2023-08-26T13:30:55.563 INFO:teuthology.orchestra.run.smithi031.stdout: osd debug verify stray on activate = true 2023-08-26T13:30:55.563 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-08-26T13:30:55.563 INFO:teuthology.orchestra.run.smithi031.stdout: osd open classes on start = true 2023-08-26T13:30:55.563 INFO:teuthology.orchestra.run.smithi031.stdout: osd debug pg log writeout = true 2023-08-26T13:30:55.563 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-08-26T13:30:55.564 INFO:teuthology.orchestra.run.smithi031.stdout: osd deep scrub update digest min age = 30 2023-08-26T13:30:55.564 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-08-26T13:30:55.564 INFO:teuthology.orchestra.run.smithi031.stdout: osd map max advance = 10 2023-08-26T13:30:55.564 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-08-26T13:30:55.564 INFO:teuthology.orchestra.run.smithi031.stdout: journal zero on create = true 2023-08-26T13:30:55.564 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-08-26T13:30:55.565 INFO:teuthology.orchestra.run.smithi031.stdout: filestore ondisk finisher threads = 3 2023-08-26T13:30:55.565 INFO:teuthology.orchestra.run.smithi031.stdout: filestore apply finisher threads = 3 2023-08-26T13:30:55.565 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-08-26T13:30:55.565 INFO:teuthology.orchestra.run.smithi031.stdout: bdev debug aio = true 2023-08-26T13:30:55.565 INFO:teuthology.orchestra.run.smithi031.stdout: osd debug misdirected ops = true 2023-08-26T13:30:55.566 INFO:teuthology.orchestra.run.smithi031.stdout: bdev async discard = True 2023-08-26T13:30:55.566 INFO:teuthology.orchestra.run.smithi031.stdout: bdev enable discard = True 2023-08-26T13:30:55.566 INFO:teuthology.orchestra.run.smithi031.stdout: bluestore allocator = stupid 2023-08-26T13:30:55.566 INFO:teuthology.orchestra.run.smithi031.stdout: bluestore block size = 96636764160 2023-08-26T13:30:55.566 INFO:teuthology.orchestra.run.smithi031.stdout: bluestore fsck on mount = True 2023-08-26T13:30:55.566 INFO:teuthology.orchestra.run.smithi031.stdout: debug bluefs = 1/20 2023-08-26T13:30:55.567 INFO:teuthology.orchestra.run.smithi031.stdout: debug bluestore = 1/20 2023-08-26T13:30:55.567 INFO:teuthology.orchestra.run.smithi031.stdout: debug ms = 1 2023-08-26T13:30:55.567 INFO:teuthology.orchestra.run.smithi031.stdout: debug osd = 20 2023-08-26T13:30:55.567 INFO:teuthology.orchestra.run.smithi031.stdout: debug rocksdb = 4/10 2023-08-26T13:30:55.567 INFO:teuthology.orchestra.run.smithi031.stdout: mon osd backfillfull_ratio = 0.85 2023-08-26T13:30:55.568 INFO:teuthology.orchestra.run.smithi031.stdout: mon osd full ratio = 0.9 2023-08-26T13:30:55.568 INFO:teuthology.orchestra.run.smithi031.stdout: mon osd nearfull ratio = 0.8 2023-08-26T13:30:55.568 INFO:teuthology.orchestra.run.smithi031.stdout: osd failsafe full ratio = 0.95 2023-08-26T13:30:55.568 INFO:teuthology.orchestra.run.smithi031.stdout: osd objectstore = bluestore 2023-08-26T13:30:55.568 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-08-26T13:30:55.568 INFO:teuthology.orchestra.run.smithi031.stdout:[mgr] 2023-08-26T13:30:55.569 INFO:teuthology.orchestra.run.smithi031.stdout: debug ms = 1 2023-08-26T13:30:55.569 INFO:teuthology.orchestra.run.smithi031.stdout: debug mgr = 20 2023-08-26T13:30:55.569 INFO:teuthology.orchestra.run.smithi031.stdout: debug mon = 20 2023-08-26T13:30:55.569 INFO:teuthology.orchestra.run.smithi031.stdout: debug auth = 20 2023-08-26T13:30:55.569 INFO:teuthology.orchestra.run.smithi031.stdout: mon reweight min pgs per osd = 4 2023-08-26T13:30:55.570 INFO:teuthology.orchestra.run.smithi031.stdout: mon reweight min bytes per osd = 10 2023-08-26T13:30:55.570 INFO:teuthology.orchestra.run.smithi031.stdout: mgr/telemetry/nag = false 2023-08-26T13:30:55.570 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-08-26T13:30:55.570 INFO:teuthology.orchestra.run.smithi031.stdout:[mon] 2023-08-26T13:30:55.570 INFO:teuthology.orchestra.run.smithi031.stdout: debug ms = 1 2023-08-26T13:30:55.570 INFO:teuthology.orchestra.run.smithi031.stdout: debug mon = 20 2023-08-26T13:30:55.571 INFO:teuthology.orchestra.run.smithi031.stdout: debug paxos = 20 2023-08-26T13:30:55.571 INFO:teuthology.orchestra.run.smithi031.stdout: debug auth = 20 2023-08-26T13:30:55.571 INFO:teuthology.orchestra.run.smithi031.stdout: mon data avail warn = 5 2023-08-26T13:30:55.571 INFO:teuthology.orchestra.run.smithi031.stdout: mon mgr mkfs grace = 240 2023-08-26T13:30:55.572 INFO:teuthology.orchestra.run.smithi031.stdout: mon reweight min pgs per osd = 4 2023-08-26T13:30:55.572 INFO:teuthology.orchestra.run.smithi031.stdout: mon osd reporter subtree level = osd 2023-08-26T13:30:55.572 INFO:teuthology.orchestra.run.smithi031.stdout: mon osd prime pg temp = true 2023-08-26T13:30:55.572 INFO:teuthology.orchestra.run.smithi031.stdout: mon reweight min bytes per osd = 10 2023-08-26T13:30:55.572 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-08-26T13:30:55.573 INFO:teuthology.orchestra.run.smithi031.stdout: # rotate auth tickets quickly to exercise renewal paths 2023-08-26T13:30:55.573 INFO:teuthology.orchestra.run.smithi031.stdout: auth mon ticket ttl = 660 # 11m 2023-08-26T13:30:55.573 INFO:teuthology.orchestra.run.smithi031.stdout: auth service ticket ttl = 240 # 4m 2023-08-26T13:30:55.573 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-08-26T13:30:55.573 INFO:teuthology.orchestra.run.smithi031.stdout: # don't complain about insecure global_id in the test suite 2023-08-26T13:30:55.573 INFO:teuthology.orchestra.run.smithi031.stdout: mon_warn_on_insecure_global_id_reclaim = false 2023-08-26T13:30:55.574 INFO:teuthology.orchestra.run.smithi031.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2023-08-26T13:30:55.574 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-08-26T13:30:55.574 INFO:teuthology.orchestra.run.smithi031.stdout: # 1m isn't quite enough 2023-08-26T13:30:55.574 INFO:teuthology.orchestra.run.smithi031.stdout: mon_down_mkfs_grace = 2m 2023-08-26T13:30:55.574 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-08-26T13:30:55.575 INFO:teuthology.orchestra.run.smithi031.stdout: mon_warn_on_filestore_osds = false 2023-08-26T13:30:55.575 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-08-26T13:30:55.575 INFO:teuthology.orchestra.run.smithi031.stdout:[client] 2023-08-26T13:30:55.575 INFO:teuthology.orchestra.run.smithi031.stdout: rgw cache enabled = true 2023-08-26T13:30:55.575 INFO:teuthology.orchestra.run.smithi031.stdout: rgw enable ops log = true 2023-08-26T13:30:55.576 INFO:teuthology.orchestra.run.smithi031.stdout: rgw enable usage log = true 2023-08-26T13:30:55.576 INFO:teuthology.orchestra.run.smithi031.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2023-08-26T13:30:55.576 INFO:teuthology.orchestra.run.smithi031.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2023-08-26T13:30:55.576 INFO:teuthology.orchestra.run.smithi031.stdout: debug rbd = 20 2023-08-26T13:30:55.576 INFO:teuthology.orchestra.run.smithi031.stdout: debug rbd_mirror = 15 2023-08-26T13:30:55.576 INFO:teuthology.orchestra.run.smithi031.stdout: log to stderr = False 2023-08-26T13:30:55.577 INFO:teuthology.orchestra.run.smithi031.stdout: rbd default features = 125 2023-08-26T13:30:55.577 INFO:teuthology.orchestra.run.smithi031.stdout: rbd mirror image policy type = simple 2023-08-26T13:30:55.577 INFO:teuthology.orchestra.run.smithi031.stdout:[mon.a] 2023-08-26T13:30:55.577 INFO:teuthology.orchestra.run.smithi031.stdout:[client.mirror.0] 2023-08-26T13:30:55.577 INFO:teuthology.orchestra.run.smithi031.stdout: admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2023-08-26T13:30:55.577 INFO:teuthology.orchestra.run.smithi031.stdout: pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2023-08-26T13:30:55.578 INFO:teuthology.orchestra.run.smithi031.stdout:[client.mirror.1] 2023-08-26T13:30:55.578 INFO:teuthology.orchestra.run.smithi031.stdout: admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2023-08-26T13:30:55.578 INFO:teuthology.orchestra.run.smithi031.stdout: pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2023-08-26T13:30:55.578 INFO:teuthology.orchestra.run.smithi031.stdout:[client.mirror.2] 2023-08-26T13:30:55.578 INFO:teuthology.orchestra.run.smithi031.stdout: admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2023-08-26T13:30:55.579 INFO:teuthology.orchestra.run.smithi031.stdout: pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2023-08-26T13:30:55.579 INFO:teuthology.orchestra.run.smithi031.stdout:[client.mirror.3] 2023-08-26T13:30:55.579 INFO:teuthology.orchestra.run.smithi031.stdout: admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2023-08-26T13:30:55.579 INFO:teuthology.orchestra.run.smithi031.stdout: pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2023-08-26T13:30:55.581 INFO:teuthology.orchestra.run.smithi124.stdout:[global] 2023-08-26T13:30:55.581 INFO:teuthology.orchestra.run.smithi124.stdout: chdir = "" 2023-08-26T13:30:55.581 INFO:teuthology.orchestra.run.smithi124.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2023-08-26T13:30:55.581 INFO:teuthology.orchestra.run.smithi124.stdout: auth supported = cephx 2023-08-26T13:30:55.581 INFO:teuthology.orchestra.run.smithi124.stdout: 2023-08-26T13:30:55.582 INFO:teuthology.orchestra.run.smithi124.stdout: filestore xattr use omap = true 2023-08-26T13:30:55.582 INFO:teuthology.orchestra.run.smithi124.stdout: 2023-08-26T13:30:55.582 INFO:teuthology.orchestra.run.smithi124.stdout: mon clock drift allowed = 1.000 2023-08-26T13:30:55.582 INFO:teuthology.orchestra.run.smithi124.stdout: 2023-08-26T13:30:55.582 INFO:teuthology.orchestra.run.smithi124.stdout: osd crush chooseleaf type = 0 2023-08-26T13:30:55.583 INFO:teuthology.orchestra.run.smithi124.stdout: auth debug = true 2023-08-26T13:30:55.583 INFO:teuthology.orchestra.run.smithi124.stdout: 2023-08-26T13:30:55.583 INFO:teuthology.orchestra.run.smithi124.stdout: ms die on old message = true 2023-08-26T13:30:55.583 INFO:teuthology.orchestra.run.smithi124.stdout: ms die on bug = true 2023-08-26T13:30:55.583 INFO:teuthology.orchestra.run.smithi124.stdout: 2023-08-26T13:30:55.584 INFO:teuthology.orchestra.run.smithi124.stdout: mon max pg per osd = 10000 # >= luminous 2023-08-26T13:30:55.584 INFO:teuthology.orchestra.run.smithi124.stdout: mon pg warn max object skew = 0 2023-08-26T13:30:55.584 INFO:teuthology.orchestra.run.smithi124.stdout: 2023-08-26T13:30:55.584 INFO:teuthology.orchestra.run.smithi124.stdout: # disable pg_autoscaler by default for new pools 2023-08-26T13:30:55.584 INFO:teuthology.orchestra.run.smithi124.stdout: osd_pool_default_pg_autoscale_mode = off 2023-08-26T13:30:55.585 INFO:teuthology.orchestra.run.smithi124.stdout: 2023-08-26T13:30:55.585 INFO:teuthology.orchestra.run.smithi124.stdout: osd pool default size = 2 2023-08-26T13:30:55.585 INFO:teuthology.orchestra.run.smithi124.stdout: 2023-08-26T13:30:55.585 INFO:teuthology.orchestra.run.smithi124.stdout: mon osd allow primary affinity = true 2023-08-26T13:30:55.585 INFO:teuthology.orchestra.run.smithi124.stdout: mon osd allow pg remap = true 2023-08-26T13:30:55.585 INFO:teuthology.orchestra.run.smithi124.stdout: mon warn on legacy crush tunables = false 2023-08-26T13:30:55.586 INFO:teuthology.orchestra.run.smithi124.stdout: mon warn on crush straw calc version zero = false 2023-08-26T13:30:55.586 INFO:teuthology.orchestra.run.smithi124.stdout: mon warn on no sortbitwise = false 2023-08-26T13:30:55.586 INFO:teuthology.orchestra.run.smithi124.stdout: mon warn on osd down out interval zero = false 2023-08-26T13:30:55.586 INFO:teuthology.orchestra.run.smithi124.stdout: mon warn on too few osds = false 2023-08-26T13:30:55.586 INFO:teuthology.orchestra.run.smithi124.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2023-08-26T13:30:55.586 INFO:teuthology.orchestra.run.smithi124.stdout: mon_warn_on_pool_no_redundancy = false 2023-08-26T13:30:55.587 INFO:teuthology.orchestra.run.smithi124.stdout: mon_allow_pool_size_one = true 2023-08-26T13:30:55.587 INFO:teuthology.orchestra.run.smithi124.stdout: 2023-08-26T13:30:55.587 INFO:teuthology.orchestra.run.smithi124.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2023-08-26T13:30:55.587 INFO:teuthology.orchestra.run.smithi124.stdout: 2023-08-26T13:30:55.587 INFO:teuthology.orchestra.run.smithi124.stdout: osd default data pool replay window = 5 2023-08-26T13:30:55.588 INFO:teuthology.orchestra.run.smithi124.stdout: 2023-08-26T13:30:55.588 INFO:teuthology.orchestra.run.smithi124.stdout: mon allow pool delete = true 2023-08-26T13:30:55.588 INFO:teuthology.orchestra.run.smithi124.stdout: 2023-08-26T13:30:55.588 INFO:teuthology.orchestra.run.smithi124.stdout: mon cluster log file level = debug 2023-08-26T13:30:55.588 INFO:teuthology.orchestra.run.smithi124.stdout: debug asserts on shutdown = true 2023-08-26T13:30:55.588 INFO:teuthology.orchestra.run.smithi124.stdout: mon health detail to clog = false 2023-08-26T13:30:55.589 INFO:teuthology.orchestra.run.smithi124.stdout: mon host = 172.21.15.31 2023-08-26T13:30:55.589 INFO:teuthology.orchestra.run.smithi124.stdout: mon client directed command retry = 5 2023-08-26T13:30:55.589 INFO:teuthology.orchestra.run.smithi124.stdout: ms inject socket failures = 5000 2023-08-26T13:30:55.589 INFO:teuthology.orchestra.run.smithi124.stdout: fsid = f17edf88-b4bf-49db-bf48-f12793acdc0b 2023-08-26T13:30:55.589 INFO:teuthology.orchestra.run.smithi124.stdout: 2023-08-26T13:30:55.590 INFO:teuthology.orchestra.run.smithi124.stdout:[osd] 2023-08-26T13:30:55.590 INFO:teuthology.orchestra.run.smithi124.stdout: osd journal size = 100 2023-08-26T13:30:55.590 INFO:teuthology.orchestra.run.smithi124.stdout: 2023-08-26T13:30:55.590 INFO:teuthology.orchestra.run.smithi124.stdout: osd scrub load threshold = 5.0 2023-08-26T13:30:55.590 INFO:teuthology.orchestra.run.smithi124.stdout: osd scrub max interval = 600 2023-08-26T13:30:55.590 INFO:teuthology.orchestra.run.smithi124.stdout: osd mclock profile = high_recovery_ops 2023-08-26T13:30:55.591 INFO:teuthology.orchestra.run.smithi124.stdout: 2023-08-26T13:30:55.591 INFO:teuthology.orchestra.run.smithi124.stdout: osd recover clone overlap = true 2023-08-26T13:30:55.591 INFO:teuthology.orchestra.run.smithi124.stdout: osd recovery max chunk = 1048576 2023-08-26T13:30:55.591 INFO:teuthology.orchestra.run.smithi124.stdout: 2023-08-26T13:30:55.591 INFO:teuthology.orchestra.run.smithi124.stdout: osd debug shutdown = true 2023-08-26T13:30:55.592 INFO:teuthology.orchestra.run.smithi124.stdout: osd debug op order = true 2023-08-26T13:30:55.592 INFO:teuthology.orchestra.run.smithi124.stdout: osd debug verify stray on activate = true 2023-08-26T13:30:55.592 INFO:teuthology.orchestra.run.smithi124.stdout: 2023-08-26T13:30:55.592 INFO:teuthology.orchestra.run.smithi124.stdout: osd open classes on start = true 2023-08-26T13:30:55.592 INFO:teuthology.orchestra.run.smithi124.stdout: osd debug pg log writeout = true 2023-08-26T13:30:55.592 INFO:teuthology.orchestra.run.smithi124.stdout: 2023-08-26T13:30:55.593 INFO:teuthology.orchestra.run.smithi124.stdout: osd deep scrub update digest min age = 30 2023-08-26T13:30:55.593 INFO:teuthology.orchestra.run.smithi124.stdout: 2023-08-26T13:30:55.593 INFO:teuthology.orchestra.run.smithi124.stdout: osd map max advance = 10 2023-08-26T13:30:55.593 INFO:teuthology.orchestra.run.smithi124.stdout: 2023-08-26T13:30:55.593 INFO:teuthology.orchestra.run.smithi124.stdout: journal zero on create = true 2023-08-26T13:30:55.594 INFO:teuthology.orchestra.run.smithi124.stdout: 2023-08-26T13:30:55.594 INFO:teuthology.orchestra.run.smithi124.stdout: filestore ondisk finisher threads = 3 2023-08-26T13:30:55.594 INFO:teuthology.orchestra.run.smithi124.stdout: filestore apply finisher threads = 3 2023-08-26T13:30:55.594 INFO:teuthology.orchestra.run.smithi124.stdout: 2023-08-26T13:30:55.594 INFO:teuthology.orchestra.run.smithi124.stdout: bdev debug aio = true 2023-08-26T13:30:55.594 INFO:teuthology.orchestra.run.smithi124.stdout: osd debug misdirected ops = true 2023-08-26T13:30:55.595 INFO:teuthology.orchestra.run.smithi124.stdout: bdev async discard = True 2023-08-26T13:30:55.595 INFO:teuthology.orchestra.run.smithi124.stdout: bdev enable discard = True 2023-08-26T13:30:55.595 INFO:teuthology.orchestra.run.smithi124.stdout: bluestore allocator = stupid 2023-08-26T13:30:55.595 INFO:teuthology.orchestra.run.smithi124.stdout: bluestore block size = 96636764160 2023-08-26T13:30:55.595 INFO:teuthology.orchestra.run.smithi124.stdout: bluestore fsck on mount = True 2023-08-26T13:30:55.596 INFO:teuthology.orchestra.run.smithi124.stdout: debug bluefs = 1/20 2023-08-26T13:30:55.596 INFO:teuthology.orchestra.run.smithi124.stdout: debug bluestore = 1/20 2023-08-26T13:30:55.596 INFO:teuthology.orchestra.run.smithi124.stdout: debug ms = 1 2023-08-26T13:30:55.596 INFO:teuthology.orchestra.run.smithi124.stdout: debug osd = 20 2023-08-26T13:30:55.596 INFO:teuthology.orchestra.run.smithi124.stdout: debug rocksdb = 4/10 2023-08-26T13:30:55.596 INFO:teuthology.orchestra.run.smithi124.stdout: mon osd backfillfull_ratio = 0.85 2023-08-26T13:30:55.597 INFO:teuthology.orchestra.run.smithi124.stdout: mon osd full ratio = 0.9 2023-08-26T13:30:55.597 INFO:teuthology.orchestra.run.smithi124.stdout: mon osd nearfull ratio = 0.8 2023-08-26T13:30:55.597 INFO:teuthology.orchestra.run.smithi124.stdout: osd failsafe full ratio = 0.95 2023-08-26T13:30:55.597 INFO:teuthology.orchestra.run.smithi124.stdout: osd objectstore = bluestore 2023-08-26T13:30:55.597 INFO:teuthology.orchestra.run.smithi124.stdout: 2023-08-26T13:30:55.597 INFO:teuthology.orchestra.run.smithi124.stdout:[mgr] 2023-08-26T13:30:55.598 INFO:teuthology.orchestra.run.smithi124.stdout: debug ms = 1 2023-08-26T13:30:55.598 INFO:teuthology.orchestra.run.smithi124.stdout: debug mgr = 20 2023-08-26T13:30:55.598 INFO:teuthology.orchestra.run.smithi124.stdout: debug mon = 20 2023-08-26T13:30:55.598 INFO:teuthology.orchestra.run.smithi124.stdout: debug auth = 20 2023-08-26T13:30:55.598 INFO:teuthology.orchestra.run.smithi124.stdout: mon reweight min pgs per osd = 4 2023-08-26T13:30:55.598 INFO:teuthology.orchestra.run.smithi124.stdout: mon reweight min bytes per osd = 10 2023-08-26T13:30:55.599 INFO:teuthology.orchestra.run.smithi124.stdout: mgr/telemetry/nag = false 2023-08-26T13:30:55.599 INFO:teuthology.orchestra.run.smithi124.stdout: 2023-08-26T13:30:55.599 INFO:teuthology.orchestra.run.smithi124.stdout:[mon] 2023-08-26T13:30:55.599 INFO:teuthology.orchestra.run.smithi124.stdout: debug ms = 1 2023-08-26T13:30:55.599 INFO:teuthology.orchestra.run.smithi124.stdout: debug mon = 20 2023-08-26T13:30:55.600 INFO:teuthology.orchestra.run.smithi124.stdout: debug paxos = 20 2023-08-26T13:30:55.600 INFO:teuthology.orchestra.run.smithi124.stdout: debug auth = 20 2023-08-26T13:30:55.600 INFO:teuthology.orchestra.run.smithi124.stdout: mon data avail warn = 5 2023-08-26T13:30:55.600 INFO:teuthology.orchestra.run.smithi124.stdout: mon mgr mkfs grace = 240 2023-08-26T13:30:55.600 INFO:teuthology.orchestra.run.smithi124.stdout: mon reweight min pgs per osd = 4 2023-08-26T13:30:55.600 INFO:teuthology.orchestra.run.smithi124.stdout: mon osd reporter subtree level = osd 2023-08-26T13:30:55.601 INFO:teuthology.orchestra.run.smithi124.stdout: mon osd prime pg temp = true 2023-08-26T13:30:55.601 INFO:teuthology.orchestra.run.smithi124.stdout: mon reweight min bytes per osd = 10 2023-08-26T13:30:55.601 INFO:teuthology.orchestra.run.smithi124.stdout: 2023-08-26T13:30:55.601 INFO:teuthology.orchestra.run.smithi124.stdout: # rotate auth tickets quickly to exercise renewal paths 2023-08-26T13:30:55.601 INFO:teuthology.orchestra.run.smithi124.stdout: auth mon ticket ttl = 660 # 11m 2023-08-26T13:30:55.602 INFO:teuthology.orchestra.run.smithi124.stdout: auth service ticket ttl = 240 # 4m 2023-08-26T13:30:55.602 INFO:teuthology.orchestra.run.smithi124.stdout: 2023-08-26T13:30:55.602 INFO:teuthology.orchestra.run.smithi124.stdout: # don't complain about insecure global_id in the test suite 2023-08-26T13:30:55.602 INFO:teuthology.orchestra.run.smithi124.stdout: mon_warn_on_insecure_global_id_reclaim = false 2023-08-26T13:30:55.602 INFO:teuthology.orchestra.run.smithi124.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2023-08-26T13:30:55.603 INFO:teuthology.orchestra.run.smithi124.stdout: 2023-08-26T13:30:55.603 INFO:teuthology.orchestra.run.smithi124.stdout: # 1m isn't quite enough 2023-08-26T13:30:55.603 INFO:teuthology.orchestra.run.smithi124.stdout: mon_down_mkfs_grace = 2m 2023-08-26T13:30:55.603 INFO:teuthology.orchestra.run.smithi124.stdout: 2023-08-26T13:30:55.603 INFO:teuthology.orchestra.run.smithi124.stdout: mon_warn_on_filestore_osds = false 2023-08-26T13:30:55.603 INFO:teuthology.orchestra.run.smithi124.stdout: 2023-08-26T13:30:55.604 INFO:teuthology.orchestra.run.smithi124.stdout:[client] 2023-08-26T13:30:55.604 INFO:teuthology.orchestra.run.smithi124.stdout: rgw cache enabled = true 2023-08-26T13:30:55.604 INFO:teuthology.orchestra.run.smithi124.stdout: rgw enable ops log = true 2023-08-26T13:30:55.604 INFO:teuthology.orchestra.run.smithi124.stdout: rgw enable usage log = true 2023-08-26T13:30:55.604 INFO:teuthology.orchestra.run.smithi124.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2023-08-26T13:30:55.604 INFO:teuthology.orchestra.run.smithi124.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2023-08-26T13:30:55.605 INFO:teuthology.orchestra.run.smithi124.stdout: debug rbd = 20 2023-08-26T13:30:55.605 INFO:teuthology.orchestra.run.smithi124.stdout: debug rbd_mirror = 15 2023-08-26T13:30:55.605 INFO:teuthology.orchestra.run.smithi124.stdout: log to stderr = False 2023-08-26T13:30:55.605 INFO:teuthology.orchestra.run.smithi124.stdout: rbd default features = 125 2023-08-26T13:30:55.605 INFO:teuthology.orchestra.run.smithi124.stdout: rbd mirror image policy type = simple 2023-08-26T13:30:55.606 INFO:teuthology.orchestra.run.smithi124.stdout:[mon.a] 2023-08-26T13:30:55.606 INFO:teuthology.orchestra.run.smithi124.stdout:[client.mirror.0] 2023-08-26T13:30:55.606 INFO:teuthology.orchestra.run.smithi124.stdout: admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2023-08-26T13:30:55.606 INFO:teuthology.orchestra.run.smithi124.stdout: pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2023-08-26T13:30:55.606 INFO:teuthology.orchestra.run.smithi124.stdout:[client.mirror.1] 2023-08-26T13:30:55.606 INFO:teuthology.orchestra.run.smithi124.stdout: admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2023-08-26T13:30:55.607 INFO:teuthology.orchestra.run.smithi124.stdout: pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2023-08-26T13:30:55.607 INFO:teuthology.orchestra.run.smithi124.stdout:[client.mirror.2] 2023-08-26T13:30:55.607 INFO:teuthology.orchestra.run.smithi124.stdout: admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2023-08-26T13:30:55.607 INFO:teuthology.orchestra.run.smithi124.stdout: pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2023-08-26T13:30:55.607 INFO:teuthology.orchestra.run.smithi124.stdout:[client.mirror.3] 2023-08-26T13:30:55.608 INFO:teuthology.orchestra.run.smithi124.stdout: admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2023-08-26T13:30:55.608 INFO:teuthology.orchestra.run.smithi124.stdout: pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2023-08-26T13:30:55.608 INFO:tasks.ceph:Creating admin key on cluster1.mon.a... 2023-08-26T13:30:55.608 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=client.admin --cap mon 'allow *' --cap osd 'allow *' --cap mds 'allow *' --cap mgr 'allow *' /etc/ceph/cluster1.keyring 2023-08-26T13:30:55.679 INFO:tasks.ceph:Copying monmap to all nodes... 2023-08-26T13:30:55.680 DEBUG:teuthology.orchestra.run.smithi031:> set -ex 2023-08-26T13:30:55.680 DEBUG:teuthology.orchestra.run.smithi031:> dd if=/etc/ceph/cluster1.keyring of=/dev/stdout 2023-08-26T13:30:55.697 DEBUG:teuthology.orchestra.run.smithi031:> set -ex 2023-08-26T13:30:55.697 DEBUG:teuthology.orchestra.run.smithi031:> dd if=/home/ubuntu/cephtest/cluster1.monmap of=/dev/stdout 2023-08-26T13:30:55.753 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi031.front.sepia.ceph.com 2023-08-26T13:30:55.754 DEBUG:teuthology.orchestra.run.smithi031:> set -ex 2023-08-26T13:30:55.754 DEBUG:teuthology.orchestra.run.smithi031:> sudo dd of=/etc/ceph/cluster1.keyring 2023-08-26T13:30:55.754 DEBUG:teuthology.orchestra.run.smithi031:> sudo chmod 0644 /etc/ceph/cluster1.keyring 2023-08-26T13:30:55.831 DEBUG:teuthology.orchestra.run.smithi031:> set -ex 2023-08-26T13:30:55.832 DEBUG:teuthology.orchestra.run.smithi031:> dd of=/home/ubuntu/cephtest/cluster1.monmap 2023-08-26T13:30:55.889 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi124.front.sepia.ceph.com 2023-08-26T13:30:55.889 DEBUG:teuthology.orchestra.run.smithi124:> set -ex 2023-08-26T13:30:55.890 DEBUG:teuthology.orchestra.run.smithi124:> sudo dd of=/etc/ceph/cluster1.keyring 2023-08-26T13:30:55.890 DEBUG:teuthology.orchestra.run.smithi124:> sudo chmod 0644 /etc/ceph/cluster1.keyring 2023-08-26T13:30:55.935 DEBUG:teuthology.orchestra.run.smithi124:> set -ex 2023-08-26T13:30:55.935 DEBUG:teuthology.orchestra.run.smithi124:> dd of=/home/ubuntu/cephtest/cluster1.monmap 2023-08-26T13:30:55.995 INFO:tasks.ceph:Setting up mon nodes... 2023-08-26T13:30:55.995 INFO:tasks.ceph:Setting up mgr nodes... 2023-08-26T13:30:55.996 DEBUG:teuthology.orchestra.run.smithi031:> sudo mkdir -p /var/lib/ceph/mgr/cluster1-x && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.x /var/lib/ceph/mgr/cluster1-x/keyring 2023-08-26T13:30:56.052 INFO:teuthology.orchestra.run.smithi031.stdout:creating /var/lib/ceph/mgr/cluster1-x/keyring 2023-08-26T13:30:56.054 INFO:tasks.ceph:Setting up mds nodes... 2023-08-26T13:30:56.054 INFO:tasks.ceph_client:Setting up client nodes... 2023-08-26T13:30:56.055 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.0 /etc/ceph/cluster1.client.0.keyring && sudo chmod 0644 /etc/ceph/cluster1.client.0.keyring 2023-08-26T13:30:56.099 INFO:teuthology.orchestra.run.smithi031.stdout:creating /etc/ceph/cluster1.client.0.keyring 2023-08-26T13:30:56.113 DEBUG:teuthology.orchestra.run.smithi124:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.mirror /etc/ceph/cluster1.client.mirror.keyring && sudo chmod 0644 /etc/ceph/cluster1.client.mirror.keyring 2023-08-26T13:30:56.174 INFO:teuthology.orchestra.run.smithi124.stdout:creating /etc/ceph/cluster1.client.mirror.keyring 2023-08-26T13:30:56.189 DEBUG:teuthology.orchestra.run.smithi124:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.mirror.0 /etc/ceph/cluster1.client.mirror.0.keyring && sudo chmod 0644 /etc/ceph/cluster1.client.mirror.0.keyring 2023-08-26T13:30:56.236 INFO:teuthology.orchestra.run.smithi124.stdout:creating /etc/ceph/cluster1.client.mirror.0.keyring 2023-08-26T13:30:56.249 DEBUG:teuthology.orchestra.run.smithi124:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.mirror.1 /etc/ceph/cluster1.client.mirror.1.keyring && sudo chmod 0644 /etc/ceph/cluster1.client.mirror.1.keyring 2023-08-26T13:30:56.292 INFO:teuthology.orchestra.run.smithi124.stdout:creating /etc/ceph/cluster1.client.mirror.1.keyring 2023-08-26T13:30:56.306 DEBUG:teuthology.orchestra.run.smithi124:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.mirror.2 /etc/ceph/cluster1.client.mirror.2.keyring && sudo chmod 0644 /etc/ceph/cluster1.client.mirror.2.keyring 2023-08-26T13:30:56.388 INFO:teuthology.orchestra.run.smithi124.stdout:creating /etc/ceph/cluster1.client.mirror.2.keyring 2023-08-26T13:30:56.401 DEBUG:teuthology.orchestra.run.smithi124:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.mirror.3 /etc/ceph/cluster1.client.mirror.3.keyring && sudo chmod 0644 /etc/ceph/cluster1.client.mirror.3.keyring 2023-08-26T13:30:56.485 INFO:teuthology.orchestra.run.smithi124.stdout:creating /etc/ceph/cluster1.client.mirror.3.keyring 2023-08-26T13:30:56.497 DEBUG:teuthology.orchestra.run.smithi124:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.mirror.4 /etc/ceph/cluster1.client.mirror.4.keyring && sudo chmod 0644 /etc/ceph/cluster1.client.mirror.4.keyring 2023-08-26T13:30:56.581 INFO:teuthology.orchestra.run.smithi124.stdout:creating /etc/ceph/cluster1.client.mirror.4.keyring 2023-08-26T13:30:56.594 DEBUG:teuthology.orchestra.run.smithi124:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.mirror.5 /etc/ceph/cluster1.client.mirror.5.keyring && sudo chmod 0644 /etc/ceph/cluster1.client.mirror.5.keyring 2023-08-26T13:30:56.676 INFO:teuthology.orchestra.run.smithi124.stdout:creating /etc/ceph/cluster1.client.mirror.5.keyring 2023-08-26T13:30:56.689 DEBUG:teuthology.orchestra.run.smithi124:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.mirror.6 /etc/ceph/cluster1.client.mirror.6.keyring && sudo chmod 0644 /etc/ceph/cluster1.client.mirror.6.keyring 2023-08-26T13:30:56.771 INFO:teuthology.orchestra.run.smithi124.stdout:creating /etc/ceph/cluster1.client.mirror.6.keyring 2023-08-26T13:30:56.784 INFO:tasks.ceph:Running mkfs on osd nodes... 2023-08-26T13:30:56.784 INFO:tasks.ceph:ctx.disk_config.remote_to_roles_to_dev: {Remote(name='ubuntu@smithi031.front.sepia.ceph.com'): {'cluster1.osd.0': '/dev/vg_nvme/lv_1', 'cluster1.osd.1': '/dev/vg_nvme/lv_2', 'cluster1.osd.2': '/dev/vg_nvme/lv_3'}} 2023-08-26T13:30:56.785 DEBUG:teuthology.orchestra.run.smithi031:> sudo mkdir -p /var/lib/ceph/osd/cluster1-0 2023-08-26T13:30:56.816 INFO:tasks.ceph:roles_to_devs: {'cluster1.osd.0': '/dev/vg_nvme/lv_1', 'cluster1.osd.1': '/dev/vg_nvme/lv_2', 'cluster1.osd.2': '/dev/vg_nvme/lv_3'} 2023-08-26T13:30:56.816 INFO:tasks.ceph:role: cluster1.osd.0 2023-08-26T13:30:56.817 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi031.front.sepia.ceph.com 2023-08-26T13:30:56.817 DEBUG:teuthology.orchestra.run.smithi031:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2023-08-26T13:30:56.894 INFO:teuthology.orchestra.run.smithi031.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2023-08-26T13:30:56.894 INFO:teuthology.orchestra.run.smithi031.stdout: = sectsz=512 attr=2, projid32bit=1 2023-08-26T13:30:56.895 INFO:teuthology.orchestra.run.smithi031.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2023-08-26T13:30:56.895 INFO:teuthology.orchestra.run.smithi031.stdout: = reflink=1 bigtime=0 inobtcount=0 2023-08-26T13:30:56.895 INFO:teuthology.orchestra.run.smithi031.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2023-08-26T13:30:56.895 INFO:teuthology.orchestra.run.smithi031.stdout: = sunit=0 swidth=0 blks 2023-08-26T13:30:56.895 INFO:teuthology.orchestra.run.smithi031.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2023-08-26T13:30:56.896 INFO:teuthology.orchestra.run.smithi031.stdout:log =internal log bsize=4096 blocks=11443, version=2 2023-08-26T13:30:56.896 INFO:teuthology.orchestra.run.smithi031.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2023-08-26T13:30:56.896 INFO:teuthology.orchestra.run.smithi031.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2023-08-26T13:30:57.323 INFO:teuthology.orchestra.run.smithi031.stdout:Discarding blocks...Done. 2023-08-26T13:30:57.325 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi031.front.sepia.ceph.com -o noatime 2023-08-26T13:30:57.325 DEBUG:teuthology.orchestra.run.smithi031:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/cluster1-0 2023-08-26T13:30:57.504 DEBUG:teuthology.orchestra.run.smithi031:> sudo /sbin/restorecon /var/lib/ceph/osd/cluster1-0 2023-08-26T13:30:57.541 DEBUG:teuthology.orchestra.run.smithi031:> sudo mkdir -p /var/lib/ceph/osd/cluster1-1 2023-08-26T13:30:57.611 INFO:tasks.ceph:roles_to_devs: {'cluster1.osd.0': '/dev/vg_nvme/lv_1', 'cluster1.osd.1': '/dev/vg_nvme/lv_2', 'cluster1.osd.2': '/dev/vg_nvme/lv_3'} 2023-08-26T13:30:57.611 INFO:tasks.ceph:role: cluster1.osd.1 2023-08-26T13:30:57.612 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi031.front.sepia.ceph.com 2023-08-26T13:30:57.612 DEBUG:teuthology.orchestra.run.smithi031:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2023-08-26T13:30:57.688 INFO:teuthology.orchestra.run.smithi031.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2023-08-26T13:30:57.688 INFO:teuthology.orchestra.run.smithi031.stdout: = sectsz=512 attr=2, projid32bit=1 2023-08-26T13:30:57.688 INFO:teuthology.orchestra.run.smithi031.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2023-08-26T13:30:57.688 INFO:teuthology.orchestra.run.smithi031.stdout: = reflink=1 bigtime=0 inobtcount=0 2023-08-26T13:30:57.689 INFO:teuthology.orchestra.run.smithi031.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2023-08-26T13:30:57.689 INFO:teuthology.orchestra.run.smithi031.stdout: = sunit=0 swidth=0 blks 2023-08-26T13:30:57.689 INFO:teuthology.orchestra.run.smithi031.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2023-08-26T13:30:57.689 INFO:teuthology.orchestra.run.smithi031.stdout:log =internal log bsize=4096 blocks=11443, version=2 2023-08-26T13:30:57.689 INFO:teuthology.orchestra.run.smithi031.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2023-08-26T13:30:57.690 INFO:teuthology.orchestra.run.smithi031.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2023-08-26T13:30:58.107 INFO:teuthology.orchestra.run.smithi031.stdout:Discarding blocks...Done. 2023-08-26T13:30:58.109 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi031.front.sepia.ceph.com -o noatime 2023-08-26T13:30:58.110 DEBUG:teuthology.orchestra.run.smithi031:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/cluster1-1 2023-08-26T13:30:58.232 DEBUG:teuthology.orchestra.run.smithi031:> sudo /sbin/restorecon /var/lib/ceph/osd/cluster1-1 2023-08-26T13:30:58.267 DEBUG:teuthology.orchestra.run.smithi031:> sudo mkdir -p /var/lib/ceph/osd/cluster1-2 2023-08-26T13:30:58.337 INFO:tasks.ceph:roles_to_devs: {'cluster1.osd.0': '/dev/vg_nvme/lv_1', 'cluster1.osd.1': '/dev/vg_nvme/lv_2', 'cluster1.osd.2': '/dev/vg_nvme/lv_3'} 2023-08-26T13:30:58.337 INFO:tasks.ceph:role: cluster1.osd.2 2023-08-26T13:30:58.337 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi031.front.sepia.ceph.com 2023-08-26T13:30:58.338 DEBUG:teuthology.orchestra.run.smithi031:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2023-08-26T13:30:58.414 INFO:teuthology.orchestra.run.smithi031.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2023-08-26T13:30:58.414 INFO:teuthology.orchestra.run.smithi031.stdout: = sectsz=512 attr=2, projid32bit=1 2023-08-26T13:30:58.414 INFO:teuthology.orchestra.run.smithi031.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2023-08-26T13:30:58.415 INFO:teuthology.orchestra.run.smithi031.stdout: = reflink=1 bigtime=0 inobtcount=0 2023-08-26T13:30:58.415 INFO:teuthology.orchestra.run.smithi031.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2023-08-26T13:30:58.415 INFO:teuthology.orchestra.run.smithi031.stdout: = sunit=0 swidth=0 blks 2023-08-26T13:30:58.415 INFO:teuthology.orchestra.run.smithi031.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2023-08-26T13:30:58.415 INFO:teuthology.orchestra.run.smithi031.stdout:log =internal log bsize=4096 blocks=11443, version=2 2023-08-26T13:30:58.416 INFO:teuthology.orchestra.run.smithi031.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2023-08-26T13:30:58.416 INFO:teuthology.orchestra.run.smithi031.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2023-08-26T13:30:58.857 INFO:teuthology.orchestra.run.smithi031.stdout:Discarding blocks...Done. 2023-08-26T13:30:58.859 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi031.front.sepia.ceph.com -o noatime 2023-08-26T13:30:58.860 DEBUG:teuthology.orchestra.run.smithi031:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/cluster1-2 2023-08-26T13:30:59.029 DEBUG:teuthology.orchestra.run.smithi031:> sudo /sbin/restorecon /var/lib/ceph/osd/cluster1-2 2023-08-26T13:30:59.068 DEBUG:teuthology.orchestra.run.smithi031:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster cluster1 --mkfs --mkkey -i 0 --monmap /home/ubuntu/cephtest/cluster1.monmap 2023-08-26T13:30:59.171 INFO:teuthology.orchestra.run.smithi031.stderr:2023-08-26T13:30:59.169+0000 7f6b7efbf640 -1 auth: error reading file: /var/lib/ceph/osd/cluster1-0/keyring: can't open /var/lib/ceph/osd/cluster1-0/keyring: (2) No such file or directory 2023-08-26T13:30:59.171 INFO:teuthology.orchestra.run.smithi031.stderr:2023-08-26T13:30:59.169+0000 7f6b7efbf640 -1 created new key in keyring /var/lib/ceph/osd/cluster1-0/keyring 2023-08-26T13:30:59.171 INFO:teuthology.orchestra.run.smithi031.stderr:2023-08-26T13:30:59.169+0000 7f6b7efbf640 -1 bluestore(/var/lib/ceph/osd/cluster1-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/cluster1-0/block: (2) No such file or directory 2023-08-26T13:30:59.172 INFO:teuthology.orchestra.run.smithi031.stderr:2023-08-26T13:30:59.169+0000 7f6b7efbf640 -1 bluestore(/var/lib/ceph/osd/cluster1-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/cluster1-0/block: (2) No such file or directory 2023-08-26T13:30:59.172 INFO:teuthology.orchestra.run.smithi031.stderr:2023-08-26T13:30:59.169+0000 7f6b7efbf640 -1 bluestore(/var/lib/ceph/osd/cluster1-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/cluster1-0/block: (2) No such file or directory 2023-08-26T13:30:59.177 INFO:teuthology.orchestra.run.smithi031.stderr:2023-08-26T13:30:59.176+0000 7f6b7efbf640 -1 bluestore(/var/lib/ceph/osd/cluster1-0) _read_fsid unparsable uuid 2023-08-26T13:31:02.041 DEBUG:teuthology.orchestra.run.smithi031:> sudo chown -R ceph:ceph /var/lib/ceph/osd/cluster1-0 2023-08-26T13:31:02.087 DEBUG:teuthology.orchestra.run.smithi031:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster cluster1 --mkfs --mkkey -i 1 --monmap /home/ubuntu/cephtest/cluster1.monmap 2023-08-26T13:31:02.140 INFO:teuthology.orchestra.run.smithi031.stderr:2023-08-26T13:31:02.137+0000 7fd2d0c73640 -1 auth: error reading file: /var/lib/ceph/osd/cluster1-1/keyring: can't open /var/lib/ceph/osd/cluster1-1/keyring: (2) No such file or directory 2023-08-26T13:31:02.141 INFO:teuthology.orchestra.run.smithi031.stderr:2023-08-26T13:31:02.138+0000 7fd2d0c73640 -1 created new key in keyring /var/lib/ceph/osd/cluster1-1/keyring 2023-08-26T13:31:02.141 INFO:teuthology.orchestra.run.smithi031.stderr:2023-08-26T13:31:02.138+0000 7fd2d0c73640 -1 bluestore(/var/lib/ceph/osd/cluster1-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/cluster1-1/block: (2) No such file or directory 2023-08-26T13:31:02.141 INFO:teuthology.orchestra.run.smithi031.stderr:2023-08-26T13:31:02.138+0000 7fd2d0c73640 -1 bluestore(/var/lib/ceph/osd/cluster1-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/cluster1-1/block: (2) No such file or directory 2023-08-26T13:31:02.141 INFO:teuthology.orchestra.run.smithi031.stderr:2023-08-26T13:31:02.138+0000 7fd2d0c73640 -1 bluestore(/var/lib/ceph/osd/cluster1-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/cluster1-1/block: (2) No such file or directory 2023-08-26T13:31:02.141 INFO:teuthology.orchestra.run.smithi031.stderr:2023-08-26T13:31:02.138+0000 7fd2d0c73640 -1 bluestore(/var/lib/ceph/osd/cluster1-1) _read_fsid unparsable uuid 2023-08-26T13:31:05.003 DEBUG:teuthology.orchestra.run.smithi031:> sudo chown -R ceph:ceph /var/lib/ceph/osd/cluster1-1 2023-08-26T13:31:05.036 DEBUG:teuthology.orchestra.run.smithi031:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster cluster1 --mkfs --mkkey -i 2 --monmap /home/ubuntu/cephtest/cluster1.monmap 2023-08-26T13:31:05.130 INFO:teuthology.orchestra.run.smithi031.stderr:2023-08-26T13:31:05.127+0000 7f6ef3d98640 -1 auth: error reading file: /var/lib/ceph/osd/cluster1-2/keyring: can't open /var/lib/ceph/osd/cluster1-2/keyring: (2) No such file or directory 2023-08-26T13:31:05.130 INFO:teuthology.orchestra.run.smithi031.stderr:2023-08-26T13:31:05.128+0000 7f6ef3d98640 -1 created new key in keyring /var/lib/ceph/osd/cluster1-2/keyring 2023-08-26T13:31:05.131 INFO:teuthology.orchestra.run.smithi031.stderr:2023-08-26T13:31:05.128+0000 7f6ef3d98640 -1 bluestore(/var/lib/ceph/osd/cluster1-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/cluster1-2/block: (2) No such file or directory 2023-08-26T13:31:05.131 INFO:teuthology.orchestra.run.smithi031.stderr:2023-08-26T13:31:05.128+0000 7f6ef3d98640 -1 bluestore(/var/lib/ceph/osd/cluster1-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/cluster1-2/block: (2) No such file or directory 2023-08-26T13:31:05.131 INFO:teuthology.orchestra.run.smithi031.stderr:2023-08-26T13:31:05.128+0000 7f6ef3d98640 -1 bluestore(/var/lib/ceph/osd/cluster1-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/cluster1-2/block: (2) No such file or directory 2023-08-26T13:31:05.132 INFO:teuthology.orchestra.run.smithi031.stderr:2023-08-26T13:31:05.128+0000 7f6ef3d98640 -1 bluestore(/var/lib/ceph/osd/cluster1-2) _read_fsid unparsable uuid 2023-08-26T13:31:07.996 DEBUG:teuthology.orchestra.run.smithi031:> sudo chown -R ceph:ceph /var/lib/ceph/osd/cluster1-2 2023-08-26T13:31:08.029 INFO:tasks.ceph:Reading keys from all nodes... 2023-08-26T13:31:08.030 DEBUG:teuthology.orchestra.run.smithi031:> set -ex 2023-08-26T13:31:08.030 DEBUG:teuthology.orchestra.run.smithi031:> sudo dd if=/var/lib/ceph/mgr/cluster1-x/keyring of=/dev/stdout 2023-08-26T13:31:08.100 DEBUG:teuthology.orchestra.run.smithi031:> set -ex 2023-08-26T13:31:08.101 DEBUG:teuthology.orchestra.run.smithi031:> sudo dd if=/var/lib/ceph/osd/cluster1-0/keyring of=/dev/stdout 2023-08-26T13:31:08.170 DEBUG:teuthology.orchestra.run.smithi031:> set -ex 2023-08-26T13:31:08.170 DEBUG:teuthology.orchestra.run.smithi031:> sudo dd if=/var/lib/ceph/osd/cluster1-1/keyring of=/dev/stdout 2023-08-26T13:31:08.239 DEBUG:teuthology.orchestra.run.smithi031:> set -ex 2023-08-26T13:31:08.240 DEBUG:teuthology.orchestra.run.smithi031:> sudo dd if=/var/lib/ceph/osd/cluster1-2/keyring of=/dev/stdout 2023-08-26T13:31:08.307 DEBUG:teuthology.orchestra.run.smithi031:> set -ex 2023-08-26T13:31:08.308 DEBUG:teuthology.orchestra.run.smithi031:> dd if=/etc/ceph/cluster1.client.0.keyring of=/dev/stdout 2023-08-26T13:31:08.366 DEBUG:teuthology.orchestra.run.smithi124:> set -ex 2023-08-26T13:31:08.366 DEBUG:teuthology.orchestra.run.smithi124:> dd if=/etc/ceph/cluster1.client.mirror.keyring of=/dev/stdout 2023-08-26T13:31:08.387 DEBUG:teuthology.orchestra.run.smithi124:> set -ex 2023-08-26T13:31:08.387 DEBUG:teuthology.orchestra.run.smithi124:> dd if=/etc/ceph/cluster1.client.mirror.0.keyring of=/dev/stdout 2023-08-26T13:31:08.447 DEBUG:teuthology.orchestra.run.smithi124:> set -ex 2023-08-26T13:31:08.447 DEBUG:teuthology.orchestra.run.smithi124:> dd if=/etc/ceph/cluster1.client.mirror.1.keyring of=/dev/stdout 2023-08-26T13:31:08.507 DEBUG:teuthology.orchestra.run.smithi124:> set -ex 2023-08-26T13:31:08.507 DEBUG:teuthology.orchestra.run.smithi124:> dd if=/etc/ceph/cluster1.client.mirror.2.keyring of=/dev/stdout 2023-08-26T13:31:08.566 DEBUG:teuthology.orchestra.run.smithi124:> set -ex 2023-08-26T13:31:08.567 DEBUG:teuthology.orchestra.run.smithi124:> dd if=/etc/ceph/cluster1.client.mirror.3.keyring of=/dev/stdout 2023-08-26T13:31:08.626 DEBUG:teuthology.orchestra.run.smithi124:> set -ex 2023-08-26T13:31:08.626 DEBUG:teuthology.orchestra.run.smithi124:> dd if=/etc/ceph/cluster1.client.mirror.4.keyring of=/dev/stdout 2023-08-26T13:31:08.685 DEBUG:teuthology.orchestra.run.smithi124:> set -ex 2023-08-26T13:31:08.685 DEBUG:teuthology.orchestra.run.smithi124:> dd if=/etc/ceph/cluster1.client.mirror.5.keyring of=/dev/stdout 2023-08-26T13:31:08.745 DEBUG:teuthology.orchestra.run.smithi124:> set -ex 2023-08-26T13:31:08.745 DEBUG:teuthology.orchestra.run.smithi124:> dd if=/etc/ceph/cluster1.client.mirror.6.keyring of=/dev/stdout 2023-08-26T13:31:08.805 INFO:tasks.ceph:Adding keys to all mons... 2023-08-26T13:31:08.806 DEBUG:teuthology.orchestra.run.smithi031:> sudo tee -a /etc/ceph/cluster1.keyring 2023-08-26T13:31:08.837 INFO:teuthology.orchestra.run.smithi031.stdout:[mgr.x] 2023-08-26T13:31:08.837 INFO:teuthology.orchestra.run.smithi031.stdout: key = AQCQ/ulk3qYVAxAAiSMxqLY0oJMGC9/XOCTAFA== 2023-08-26T13:31:08.837 INFO:teuthology.orchestra.run.smithi031.stdout:[osd.0] 2023-08-26T13:31:08.838 INFO:teuthology.orchestra.run.smithi031.stdout: key = AQCT/ulkaJgpChAAiuNvPQ7ZwX6yzmBDdBjCRA== 2023-08-26T13:31:08.838 INFO:teuthology.orchestra.run.smithi031.stdout:[osd.1] 2023-08-26T13:31:08.838 INFO:teuthology.orchestra.run.smithi031.stdout: key = AQCW/ulkPzxVCBAAp3pGkG0u4oNUHpZ8d1IGrQ== 2023-08-26T13:31:08.838 INFO:teuthology.orchestra.run.smithi031.stdout:[osd.2] 2023-08-26T13:31:08.838 INFO:teuthology.orchestra.run.smithi031.stdout: key = AQCZ/ulkzQ2uBxAA67JQWeQZQGgUpV5Tvk98aQ== 2023-08-26T13:31:08.839 INFO:teuthology.orchestra.run.smithi031.stdout:[client.0] 2023-08-26T13:31:08.839 INFO:teuthology.orchestra.run.smithi031.stdout: key = AQCQ/ulkReHpBRAA5nj2LTR3Pg+55Q3ObSX7sg== 2023-08-26T13:31:08.839 INFO:teuthology.orchestra.run.smithi031.stdout:[client.mirror] 2023-08-26T13:31:08.839 INFO:teuthology.orchestra.run.smithi031.stdout: key = AQCQ/ulkbGRoChAA/u7HuF25JkXFTfXHFlpI5w== 2023-08-26T13:31:08.839 INFO:teuthology.orchestra.run.smithi031.stdout:[client.mirror.0] 2023-08-26T13:31:08.840 INFO:teuthology.orchestra.run.smithi031.stdout: key = AQCQ/ulk+l0MDhAAKVawsGzeCVjS5ToF1AeIeg== 2023-08-26T13:31:08.840 INFO:teuthology.orchestra.run.smithi031.stdout:[client.mirror.1] 2023-08-26T13:31:08.840 INFO:teuthology.orchestra.run.smithi031.stdout: key = AQCQ/ulkdrVtERAAysNg9WClxAVKE29CrJahtw== 2023-08-26T13:31:08.840 INFO:teuthology.orchestra.run.smithi031.stdout:[client.mirror.2] 2023-08-26T13:31:08.840 INFO:teuthology.orchestra.run.smithi031.stdout: key = AQCQ/ulke08tFxAAsLvyyB7D2xHwyH/4zjNTBg== 2023-08-26T13:31:08.841 INFO:teuthology.orchestra.run.smithi031.stdout:[client.mirror.3] 2023-08-26T13:31:08.841 INFO:teuthology.orchestra.run.smithi031.stdout: key = AQCQ/ulkOQjnHBAAXyvouR+GF/42kzqtzDV7OQ== 2023-08-26T13:31:08.841 INFO:teuthology.orchestra.run.smithi031.stdout:[client.mirror.4] 2023-08-26T13:31:08.841 INFO:teuthology.orchestra.run.smithi031.stdout: key = AQCQ/ulkBHmhIhAAveqfm7nLinV4pGnlxRpVCA== 2023-08-26T13:31:08.841 INFO:teuthology.orchestra.run.smithi031.stdout:[client.mirror.5] 2023-08-26T13:31:08.842 INFO:teuthology.orchestra.run.smithi031.stdout: key = AQCQ/ulkeJpSKBAAXdSoAGKuQKUEy3EF7B6Y2w== 2023-08-26T13:31:08.842 INFO:teuthology.orchestra.run.smithi031.stdout:[client.mirror.6] 2023-08-26T13:31:08.842 INFO:teuthology.orchestra.run.smithi031.stdout: key = AQCQ/ulkV8z9LRAANfK8i4AF23XMQshSOr8apg== 2023-08-26T13:31:08.843 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster1.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2023-08-26T13:31:08.945 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster1.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-08-26T13:31:09.003 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster1.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-08-26T13:31:09.059 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster1.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-08-26T13:31:09.113 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster1.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2023-08-26T13:31:09.171 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster1.keyring --name=client.mirror --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2023-08-26T13:31:09.228 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster1.keyring --name=client.mirror.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2023-08-26T13:31:09.286 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster1.keyring --name=client.mirror.1 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2023-08-26T13:31:09.344 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster1.keyring --name=client.mirror.2 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2023-08-26T13:31:09.402 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster1.keyring --name=client.mirror.3 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2023-08-26T13:31:09.460 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster1.keyring --name=client.mirror.4 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2023-08-26T13:31:09.518 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster1.keyring --name=client.mirror.5 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2023-08-26T13:31:09.576 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster1.keyring --name=client.mirror.6 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2023-08-26T13:31:09.634 INFO:tasks.ceph:Running mkfs on mon nodes... 2023-08-26T13:31:09.634 DEBUG:teuthology.orchestra.run.smithi031:> sudo mkdir -p /var/lib/ceph/mon/cluster1-a 2023-08-26T13:31:09.662 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster cluster1 --mkfs -i a --monmap /home/ubuntu/cephtest/cluster1.monmap --keyring /etc/ceph/cluster1.keyring 2023-08-26T13:31:09.763 DEBUG:teuthology.orchestra.run.smithi031:> sudo chown -R ceph:ceph /var/lib/ceph/mon/cluster1-a 2023-08-26T13:31:09.790 DEBUG:teuthology.orchestra.run.smithi031:> rm -- /home/ubuntu/cephtest/cluster1.monmap 2023-08-26T13:31:09.847 INFO:tasks.ceph:Starting mon daemons in cluster cluster1... 2023-08-26T13:31:09.848 INFO:tasks.ceph.cluster1.mon.a:Restarting daemon 2023-08-26T13:31:09.849 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster cluster1 -i a 2023-08-26T13:31:09.890 INFO:tasks.ceph.cluster1.mon.a:Started 2023-08-26T13:31:09.891 INFO:tasks.ceph:Starting mgr daemons in cluster cluster1... 2023-08-26T13:31:09.891 INFO:tasks.ceph.cluster1.mgr.x:Restarting daemon 2023-08-26T13:31:09.892 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mgr -f --cluster cluster1 -i x 2023-08-26T13:31:09.894 INFO:tasks.ceph.cluster1.mgr.x:Started 2023-08-26T13:31:09.894 INFO:tasks.ceph:Setting crush tunables to default 2023-08-26T13:31:09.894 DEBUG:teuthology.orchestra.run.smithi031:> sudo ceph --cluster cluster1 osd crush tunables default 2023-08-26T13:31:10.127 INFO:teuthology.orchestra.run.smithi031.stderr:adjusted tunables profile to default 2023-08-26T13:31:10.138 INFO:tasks.ceph:check_enable_crimson: False 2023-08-26T13:31:10.139 INFO:tasks.ceph:Starting osd daemons in cluster cluster1... 2023-08-26T13:31:10.139 DEBUG:teuthology.orchestra.run.smithi031:> set -ex 2023-08-26T13:31:10.139 DEBUG:teuthology.orchestra.run.smithi031:> sudo dd if=/var/lib/ceph/osd/cluster1-0/fsid of=/dev/stdout 2023-08-26T13:31:10.167 DEBUG:teuthology.orchestra.run.smithi031:> set -ex 2023-08-26T13:31:10.168 DEBUG:teuthology.orchestra.run.smithi031:> sudo dd if=/var/lib/ceph/osd/cluster1-1/fsid of=/dev/stdout 2023-08-26T13:31:10.235 DEBUG:teuthology.orchestra.run.smithi031:> set -ex 2023-08-26T13:31:10.235 DEBUG:teuthology.orchestra.run.smithi031:> sudo dd if=/var/lib/ceph/osd/cluster1-2/fsid of=/dev/stdout 2023-08-26T13:31:10.293 INFO:tasks.ceph.cluster1.mgr.x.smithi031.stderr:2023-08-26T13:31:10.292+0000 7f959642f200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2023-08-26T13:31:10.303 DEBUG:teuthology.orchestra.run.smithi031:> sudo ceph --cluster cluster1 osd new 4c2e839b-e689-401b-baaa-2460364bd02b 0 2023-08-26T13:31:10.363 INFO:tasks.ceph.cluster1.mgr.x.smithi031.stderr:2023-08-26T13:31:10.362+0000 7f959642f200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2023-08-26T13:31:10.445 INFO:tasks.ceph.cluster1.mgr.x.smithi031.stderr:2023-08-26T13:31:10.443+0000 7f959642f200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2023-08-26T13:31:10.530 INFO:tasks.ceph.cluster1.mgr.x.smithi031.stderr:2023-08-26T13:31:10.529+0000 7f959642f200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2023-08-26T13:31:10.564 INFO:teuthology.orchestra.run.smithi031.stdout:0 2023-08-26T13:31:10.577 DEBUG:teuthology.orchestra.run.smithi031:> sudo ceph --cluster cluster1 osd new f1a00787-5df4-493d-b218-c65081a9fd8d 1 2023-08-26T13:31:10.685 INFO:tasks.ceph.cluster1.mgr.x.smithi031.stderr:2023-08-26T13:31:10.683+0000 7f959642f200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2023-08-26T13:31:10.853 INFO:teuthology.orchestra.run.smithi031.stdout:1 2023-08-26T13:31:10.865 DEBUG:teuthology.orchestra.run.smithi031:> sudo ceph --cluster cluster1 osd new b66ec5a2-3f38-48fb-825e-51ae0f867929 2 2023-08-26T13:31:11.104 INFO:teuthology.orchestra.run.smithi031.stdout:2 2023-08-26T13:31:11.116 INFO:tasks.ceph.cluster1.osd.0:Restarting daemon 2023-08-26T13:31:11.117 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster cluster1 -i 0 2023-08-26T13:31:11.120 INFO:tasks.ceph.cluster1.osd.0:Started 2023-08-26T13:31:11.120 INFO:tasks.ceph.cluster1.osd.1:Restarting daemon 2023-08-26T13:31:11.121 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster cluster1 -i 1 2023-08-26T13:31:11.122 INFO:tasks.ceph.cluster1.mgr.x.smithi031.stderr:2023-08-26T13:31:11.120+0000 7f959642f200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2023-08-26T13:31:11.123 INFO:tasks.ceph.cluster1.osd.1:Started 2023-08-26T13:31:11.123 INFO:tasks.ceph.cluster1.osd.2:Restarting daemon 2023-08-26T13:31:11.123 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster cluster1 -i 2 2023-08-26T13:31:11.125 INFO:tasks.ceph.cluster1.osd.2:Started 2023-08-26T13:31:11.162 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 osd dump --format=json 2023-08-26T13:31:11.262 INFO:tasks.ceph.cluster1.mgr.x.smithi031.stderr:2023-08-26T13:31:11.261+0000 7f959642f200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2023-08-26T13:31:11.359 INFO:tasks.ceph.cluster1.mgr.x.smithi031.stderr:2023-08-26T13:31:11.357+0000 7f959642f200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2023-08-26T13:31:11.386 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-08-26T13:31:11.386 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":5,"fsid":"f17edf88-b4bf-49db-bf48-f12793acdc0b","created":"2023-08-26T13:31:10.022409+0000","modified":"2023-08-26T13:31:11.100836+0000","last_up_change":"0.000000","last_in_change":"2023-08-26T13:31:11.100836+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"4c2e839b-e689-401b-baaa-2460364bd02b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"f1a00787-5df4-493d-b218-c65081a9fd8d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"b66ec5a2-3f38-48fb-825e-51ae0f867929","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-08-26T13:31:11.399 INFO:tasks.ceph.ceph_manager.cluster1:[] 2023-08-26T13:31:11.399 INFO:tasks.ceph:Waiting for OSDs to come up 2023-08-26T13:31:11.444 INFO:tasks.ceph.cluster1.mgr.x.smithi031.stderr:2023-08-26T13:31:11.443+0000 7f959642f200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2023-08-26T13:31:11.702 DEBUG:teuthology.orchestra.run.smithi031:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster cluster1 osd dump --format=json 2023-08-26T13:31:11.734 INFO:tasks.ceph.cluster1.osd.1.smithi031.stderr:2023-08-26T13:31:11.733+0000 7f7e7ec81640 -1 Falling back to public interface 2023-08-26T13:31:11.737 INFO:tasks.ceph.cluster1.osd.0.smithi031.stderr:2023-08-26T13:31:11.736+0000 7fd382067640 -1 Falling back to public interface 2023-08-26T13:31:11.746 INFO:tasks.ceph.cluster1.osd.2.smithi031.stderr:2023-08-26T13:31:11.745+0000 7ffa60436640 -1 Falling back to public interface 2023-08-26T13:31:11.899 INFO:tasks.ceph.cluster1.mgr.x.smithi031.stderr:2023-08-26T13:31:11.898+0000 7f959642f200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2023-08-26T13:31:11.902 INFO:teuthology.misc.health.smithi031.stdout: 2023-08-26T13:31:11.902 INFO:teuthology.misc.health.smithi031.stdout:{"epoch":5,"fsid":"f17edf88-b4bf-49db-bf48-f12793acdc0b","created":"2023-08-26T13:31:10.022409+0000","modified":"2023-08-26T13:31:11.100836+0000","last_up_change":"0.000000","last_in_change":"2023-08-26T13:31:11.100836+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"4c2e839b-e689-401b-baaa-2460364bd02b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"f1a00787-5df4-493d-b218-c65081a9fd8d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"b66ec5a2-3f38-48fb-825e-51ae0f867929","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-08-26T13:31:11.914 DEBUG:teuthology.misc:0 of 3 OSDs are up 2023-08-26T13:31:12.565 INFO:tasks.ceph.cluster1.mgr.x.smithi031.stderr:2023-08-26T13:31:12.564+0000 7f959642f200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2023-08-26T13:31:12.638 INFO:tasks.ceph.cluster1.mgr.x.smithi031.stderr:2023-08-26T13:31:12.637+0000 7f959642f200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2023-08-26T13:31:13.361 INFO:tasks.ceph.cluster1.mgr.x.smithi031.stderr:2023-08-26T13:31:13.360+0000 7f959642f200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2023-08-26T13:31:13.442 INFO:tasks.ceph.cluster1.mgr.x.smithi031.stderr:2023-08-26T13:31:13.441+0000 7f959642f200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2023-08-26T13:31:13.589 INFO:tasks.ceph.cluster1.mgr.x.smithi031.stderr:2023-08-26T13:31:13.588+0000 7f959642f200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2023-08-26T13:31:13.834 INFO:tasks.ceph.cluster1.mgr.x.smithi031.stderr:2023-08-26T13:31:13.833+0000 7f959642f200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2023-08-26T13:31:14.056 INFO:tasks.ceph.cluster1.mgr.x.smithi031.stderr:2023-08-26T13:31:14.054+0000 7f959642f200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2023-08-26T13:31:14.195 INFO:tasks.ceph.cluster1.mgr.x.smithi031.stderr:2023-08-26T13:31:14.193+0000 7f959642f200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2023-08-26T13:31:14.451 INFO:tasks.ceph.cluster1.mgr.x.smithi031.stderr:2023-08-26T13:31:14.450+0000 7f959642f200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2023-08-26T13:31:14.613 INFO:tasks.ceph.cluster1.mgr.x.smithi031.stderr:2023-08-26T13:31:14.611+0000 7f959642f200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2023-08-26T13:31:14.740 INFO:tasks.ceph.cluster1.mgr.x.smithi031.stderr:2023-08-26T13:31:14.739+0000 7f959642f200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2023-08-26T13:31:15.007 INFO:tasks.ceph.cluster1.osd.0.smithi031.stderr:2023-08-26T13:31:15.006+0000 7fd382067640 -1 osd.0 0 log_to_monitors true 2023-08-26T13:31:15.035 INFO:tasks.ceph.cluster1.osd.2.smithi031.stderr:2023-08-26T13:31:15.034+0000 7ffa60436640 -1 osd.2 0 log_to_monitors true 2023-08-26T13:31:15.039 INFO:tasks.ceph.cluster1.osd.1.smithi031.stderr:2023-08-26T13:31:15.038+0000 7f7e7ec81640 -1 osd.1 0 log_to_monitors true 2023-08-26T13:31:15.092 INFO:tasks.ceph.cluster1.mgr.x.smithi031.stderr:2023-08-26T13:31:15.091+0000 7f959642f200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2023-08-26T13:31:15.196 INFO:tasks.ceph.cluster1.mgr.x.smithi031.stderr:2023-08-26T13:31:15.195+0000 7f959642f200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2023-08-26T13:31:15.392 INFO:tasks.ceph.cluster1.mgr.x.smithi031.stderr:2023-08-26T13:31:15.391+0000 7f959642f200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2023-08-26T13:31:16.022 INFO:tasks.ceph.cluster1.mgr.x.smithi031.stderr:2023-08-26T13:31:16.021+0000 7f959642f200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2023-08-26T13:31:17.084 INFO:tasks.ceph.cluster1.mgr.x.smithi031.stderr:2023-08-26T13:31:17.082+0000 7f954e462700 -1 mgr.server handle_report got status from non-daemon mon.a 2023-08-26T13:31:18.217 DEBUG:teuthology.orchestra.run.smithi031:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster cluster1 osd dump --format=json 2023-08-26T13:31:18.727 INFO:teuthology.misc.health.smithi031.stdout: 2023-08-26T13:31:18.728 INFO:teuthology.misc.health.smithi031.stdout:{"epoch":7,"fsid":"f17edf88-b4bf-49db-bf48-f12793acdc0b","created":"2023-08-26T13:31:10.022409+0000","modified":"2023-08-26T13:31:16.983058+0000","last_up_change":"0.000000","last_in_change":"2023-08-26T13:31:11.100836+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"4c2e839b-e689-401b-baaa-2460364bd02b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"f1a00787-5df4-493d-b218-c65081a9fd8d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"b66ec5a2-3f38-48fb-825e-51ae0f867929","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-08-26T13:31:18.745 DEBUG:teuthology.misc:0 of 3 OSDs are up 2023-08-26T13:31:18.796 INFO:tasks.ceph.cluster1.osd.0.smithi031.stderr:2023-08-26T13:31:18.795+0000 7fd375c1e700 -1 osd.0 0 waiting for initial osdmap 2023-08-26T13:31:18.800 INFO:tasks.ceph.cluster1.osd.2.smithi031.stderr:2023-08-26T13:31:18.798+0000 7ffa557f0700 -1 osd.2 0 waiting for initial osdmap 2023-08-26T13:31:18.801 INFO:tasks.ceph.cluster1.osd.1.smithi031.stderr:2023-08-26T13:31:18.799+0000 7f7e72838700 -1 osd.1 0 waiting for initial osdmap 2023-08-26T13:31:18.812 INFO:tasks.ceph.cluster1.osd.0.smithi031.stderr:2023-08-26T13:31:18.811+0000 7fd36dff9700 -1 osd.0 7 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-08-26T13:31:18.812 INFO:tasks.ceph.cluster1.osd.1.smithi031.stderr:2023-08-26T13:31:18.811+0000 7f7e6ac13700 -1 osd.1 7 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-08-26T13:31:18.819 INFO:tasks.ceph.cluster1.osd.2.smithi031.stderr:2023-08-26T13:31:18.818+0000 7ffa4c3c8700 -1 osd.2 7 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-08-26T13:31:25.049 DEBUG:teuthology.orchestra.run.smithi031:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster cluster1 osd dump --format=json 2023-08-26T13:31:25.388 INFO:teuthology.misc.health.smithi031.stdout: 2023-08-26T13:31:25.389 INFO:teuthology.misc.health.smithi031.stdout:{"epoch":11,"fsid":"f17edf88-b4bf-49db-bf48-f12793acdc0b","created":"2023-08-26T13:31:10.022409+0000","modified":"2023-08-26T13:31:22.983276+0000","last_up_change":"2023-08-26T13:31:19.805861+0000","last_in_change":"2023-08-26T13:31:11.100836+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-08-26T13:31:20.095731+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"11","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":2.9900000095367432,"score_stable":2.9900000095367432,"optimal_score":0.67000001668930054,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"4c2e839b-e689-401b-baaa-2460364bd02b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6808","nonce":244624019},{"type":"v1","addr":"172.21.15.31:6809","nonce":244624019}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6810","nonce":244624019},{"type":"v1","addr":"172.21.15.31:6811","nonce":244624019}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6814","nonce":244624019},{"type":"v1","addr":"172.21.15.31:6815","nonce":244624019}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6812","nonce":244624019},{"type":"v1","addr":"172.21.15.31:6813","nonce":244624019}]},"public_addr":"172.21.15.31:6809/244624019","cluster_addr":"172.21.15.31:6811/244624019","heartbeat_back_addr":"172.21.15.31:6815/244624019","heartbeat_front_addr":"172.21.15.31:6813/244624019","state":["exists","up"]},{"osd":1,"uuid":"f1a00787-5df4-493d-b218-c65081a9fd8d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":9,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6800","nonce":3266254750},{"type":"v1","addr":"172.21.15.31:6801","nonce":3266254750}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6802","nonce":3266254750},{"type":"v1","addr":"172.21.15.31:6803","nonce":3266254750}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6806","nonce":3266254750},{"type":"v1","addr":"172.21.15.31:6807","nonce":3266254750}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6804","nonce":3266254750},{"type":"v1","addr":"172.21.15.31:6805","nonce":3266254750}]},"public_addr":"172.21.15.31:6801/3266254750","cluster_addr":"172.21.15.31:6803/3266254750","heartbeat_back_addr":"172.21.15.31:6807/3266254750","heartbeat_front_addr":"172.21.15.31:6805/3266254750","state":["exists","up"]},{"osd":2,"uuid":"b66ec5a2-3f38-48fb-825e-51ae0f867929","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6816","nonce":1363683839},{"type":"v1","addr":"172.21.15.31:6817","nonce":1363683839}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6818","nonce":1363683839},{"type":"v1","addr":"172.21.15.31:6819","nonce":1363683839}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6822","nonce":1363683839},{"type":"v1","addr":"172.21.15.31:6823","nonce":1363683839}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6820","nonce":1363683839},{"type":"v1","addr":"172.21.15.31:6821","nonce":1363683839}]},"public_addr":"172.21.15.31:6817/1363683839","cluster_addr":"172.21.15.31:6819/1363683839","heartbeat_back_addr":"172.21.15.31:6823/1363683839","heartbeat_front_addr":"172.21.15.31:6821/1363683839","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-08-26T13:31:16.045225+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-08-26T13:31:16.049163+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-08-26T13:31:16.034540+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-08-26T13:31:25.401 DEBUG:teuthology.misc:3 of 3 OSDs are up 2023-08-26T13:31:25.401 INFO:tasks.ceph:Creating RBD pool 2023-08-26T13:31:25.402 DEBUG:teuthology.orchestra.run.smithi031:> sudo ceph --cluster cluster1 osd pool create rbd 8 2023-08-26T13:31:26.000 INFO:teuthology.orchestra.run.smithi031.stderr:pool 'rbd' created 2023-08-26T13:31:26.013 DEBUG:teuthology.orchestra.run.smithi031:> sudo ceph --cluster cluster1 osd pool application enable rbd rbd --yes-i-really-mean-it 2023-08-26T13:31:27.002 INFO:teuthology.orchestra.run.smithi031.stderr:enabled application 'rbd' on pool 'rbd' 2023-08-26T13:31:27.019 INFO:tasks.ceph:Starting mds daemons in cluster cluster1... 2023-08-26T13:31:27.020 INFO:tasks.ceph:Waiting until cluster1 daemons up and pgs clean... 2023-08-26T13:31:27.020 INFO:tasks.ceph.ceph_manager.cluster1:waiting for mgr available 2023-08-26T13:31:27.020 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 mgr dump --format=json 2023-08-26T13:31:27.021 INFO:tasks.daemonwatchdog.daemon_watchdog:watchdog starting 2023-08-26T13:31:27.423 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-08-26T13:31:27.435 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":5,"active_gid":4101,"active_name":"x","active_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6824","nonce":887295482},{"type":"v1","addr":"172.21.15.31:6825","nonce":887295482}]},"active_addr":"172.21.15.31:6825/887295482","active_change":"2023-08-26T13:31:16.035075+0000","active_mgr_features":4540138322906710015,"available":true,"standbys":[],"modules":["iostat","nfs","restful"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","upmap"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:9.4.7","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:0.0.2","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"secs","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"noautoscale":{"name":"noautoscale","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"global autoscale flag","long_desc":"Option to turn on/off the autoscaler for all pools","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"drive_group_interval":{"name":"drive_group_interval","type":"float","level":"advanced","flags":0,"default_value":"300.0","min":"","max":"","enum_allowed":[],"desc":"interval in seconds between re-application of applied drive_groups","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{},"always_on_modules":{"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"quincy":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"reef":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"]},"last_failure_osd_epoch":0,"active_clients":[{"name":"devicehealth","addrvec":[{"type":"v2","addr":"172.21.15.31:0","nonce":1774659408}]},{"name":"libcephsqlite","addrvec":[{"type":"v2","addr":"172.21.15.31:0","nonce":2863438301}]},{"name":"rbd_support","addrvec":[{"type":"v2","addr":"172.21.15.31:0","nonce":2520918909}]},{"name":"volumes","addrvec":[{"type":"v2","addr":"172.21.15.31:0","nonce":2627029104}]}]} 2023-08-26T13:31:27.439 INFO:tasks.ceph.ceph_manager.cluster1:mgr available! 2023-08-26T13:31:27.439 INFO:tasks.ceph.ceph_manager.cluster1:waiting for all up 2023-08-26T13:31:27.440 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 osd dump --format=json 2023-08-26T13:31:27.776 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-08-26T13:31:27.776 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":13,"fsid":"f17edf88-b4bf-49db-bf48-f12793acdc0b","created":"2023-08-26T13:31:10.022409+0000","modified":"2023-08-26T13:31:26.990702+0000","last_up_change":"2023-08-26T13:31:19.805861+0000","last_in_change":"2023-08-26T13:31:11.100836+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-08-26T13:31:20.095731+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"11","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":2.9900000095367432,"score_stable":2.9900000095367432,"optimal_score":0.67000001668930054,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2023-08-26T13:31:25.764242+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"13","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"4c2e839b-e689-401b-baaa-2460364bd02b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":12,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6808","nonce":244624019},{"type":"v1","addr":"172.21.15.31:6809","nonce":244624019}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6810","nonce":244624019},{"type":"v1","addr":"172.21.15.31:6811","nonce":244624019}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6814","nonce":244624019},{"type":"v1","addr":"172.21.15.31:6815","nonce":244624019}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6812","nonce":244624019},{"type":"v1","addr":"172.21.15.31:6813","nonce":244624019}]},"public_addr":"172.21.15.31:6809/244624019","cluster_addr":"172.21.15.31:6811/244624019","heartbeat_back_addr":"172.21.15.31:6815/244624019","heartbeat_front_addr":"172.21.15.31:6813/244624019","state":["exists","up"]},{"osd":1,"uuid":"f1a00787-5df4-493d-b218-c65081a9fd8d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":12,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6800","nonce":3266254750},{"type":"v1","addr":"172.21.15.31:6801","nonce":3266254750}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6802","nonce":3266254750},{"type":"v1","addr":"172.21.15.31:6803","nonce":3266254750}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6806","nonce":3266254750},{"type":"v1","addr":"172.21.15.31:6807","nonce":3266254750}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6804","nonce":3266254750},{"type":"v1","addr":"172.21.15.31:6805","nonce":3266254750}]},"public_addr":"172.21.15.31:6801/3266254750","cluster_addr":"172.21.15.31:6803/3266254750","heartbeat_back_addr":"172.21.15.31:6807/3266254750","heartbeat_front_addr":"172.21.15.31:6805/3266254750","state":["exists","up"]},{"osd":2,"uuid":"b66ec5a2-3f38-48fb-825e-51ae0f867929","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":12,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6816","nonce":1363683839},{"type":"v1","addr":"172.21.15.31:6817","nonce":1363683839}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6818","nonce":1363683839},{"type":"v1","addr":"172.21.15.31:6819","nonce":1363683839}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6822","nonce":1363683839},{"type":"v1","addr":"172.21.15.31:6823","nonce":1363683839}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6820","nonce":1363683839},{"type":"v1","addr":"172.21.15.31:6821","nonce":1363683839}]},"public_addr":"172.21.15.31:6817/1363683839","cluster_addr":"172.21.15.31:6819/1363683839","heartbeat_back_addr":"172.21.15.31:6823/1363683839","heartbeat_front_addr":"172.21.15.31:6821/1363683839","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-08-26T13:31:16.045225+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-08-26T13:31:16.049163+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-08-26T13:31:16.034540+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-08-26T13:31:27.790 INFO:tasks.ceph.ceph_manager.cluster1:all up! 2023-08-26T13:31:27.790 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 osd dump --format=json 2023-08-26T13:31:28.125 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-08-26T13:31:28.125 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":14,"fsid":"f17edf88-b4bf-49db-bf48-f12793acdc0b","created":"2023-08-26T13:31:10.022409+0000","modified":"2023-08-26T13:31:28.001573+0000","last_up_change":"2023-08-26T13:31:19.805861+0000","last_in_change":"2023-08-26T13:31:11.100836+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-08-26T13:31:20.095731+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"11","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":2.9900000095367432,"score_stable":2.9900000095367432,"optimal_score":0.67000001668930054,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2023-08-26T13:31:25.764242+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"14","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"4c2e839b-e689-401b-baaa-2460364bd02b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":12,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6808","nonce":244624019},{"type":"v1","addr":"172.21.15.31:6809","nonce":244624019}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6810","nonce":244624019},{"type":"v1","addr":"172.21.15.31:6811","nonce":244624019}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6814","nonce":244624019},{"type":"v1","addr":"172.21.15.31:6815","nonce":244624019}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6812","nonce":244624019},{"type":"v1","addr":"172.21.15.31:6813","nonce":244624019}]},"public_addr":"172.21.15.31:6809/244624019","cluster_addr":"172.21.15.31:6811/244624019","heartbeat_back_addr":"172.21.15.31:6815/244624019","heartbeat_front_addr":"172.21.15.31:6813/244624019","state":["exists","up"]},{"osd":1,"uuid":"f1a00787-5df4-493d-b218-c65081a9fd8d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":12,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6800","nonce":3266254750},{"type":"v1","addr":"172.21.15.31:6801","nonce":3266254750}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6802","nonce":3266254750},{"type":"v1","addr":"172.21.15.31:6803","nonce":3266254750}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6806","nonce":3266254750},{"type":"v1","addr":"172.21.15.31:6807","nonce":3266254750}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6804","nonce":3266254750},{"type":"v1","addr":"172.21.15.31:6805","nonce":3266254750}]},"public_addr":"172.21.15.31:6801/3266254750","cluster_addr":"172.21.15.31:6803/3266254750","heartbeat_back_addr":"172.21.15.31:6807/3266254750","heartbeat_front_addr":"172.21.15.31:6805/3266254750","state":["exists","up"]},{"osd":2,"uuid":"b66ec5a2-3f38-48fb-825e-51ae0f867929","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":12,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6816","nonce":1363683839},{"type":"v1","addr":"172.21.15.31:6817","nonce":1363683839}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6818","nonce":1363683839},{"type":"v1","addr":"172.21.15.31:6819","nonce":1363683839}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6822","nonce":1363683839},{"type":"v1","addr":"172.21.15.31:6823","nonce":1363683839}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6820","nonce":1363683839},{"type":"v1","addr":"172.21.15.31:6821","nonce":1363683839}]},"public_addr":"172.21.15.31:6817/1363683839","cluster_addr":"172.21.15.31:6819/1363683839","heartbeat_back_addr":"172.21.15.31:6823/1363683839","heartbeat_front_addr":"172.21.15.31:6821/1363683839","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-08-26T13:31:16.045225+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-08-26T13:31:16.049163+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-08-26T13:31:16.034540+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-08-26T13:31:28.140 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 tell osd.0 flush_pg_stats 2023-08-26T13:31:28.140 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 tell osd.1 flush_pg_stats 2023-08-26T13:31:28.141 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 tell osd.2 flush_pg_stats 2023-08-26T13:31:28.314 INFO:teuthology.orchestra.run.smithi031.stdout:34359738373 2023-08-26T13:31:28.314 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 osd last-stat-seq osd.1 2023-08-26T13:31:28.315 INFO:teuthology.orchestra.run.smithi031.stdout:34359738371 2023-08-26T13:31:28.316 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 osd last-stat-seq osd.2 2023-08-26T13:31:28.316 INFO:teuthology.orchestra.run.smithi031.stdout:34359738371 2023-08-26T13:31:28.317 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 osd last-stat-seq osd.0 2023-08-26T13:31:28.709 INFO:teuthology.orchestra.run.smithi031.stdout:34359738372 2023-08-26T13:31:28.716 INFO:teuthology.orchestra.run.smithi031.stdout:34359738370 2023-08-26T13:31:28.724 INFO:tasks.ceph.ceph_manager.cluster1:need seq 34359738373 got 34359738372 for osd.1 2023-08-26T13:31:28.732 INFO:teuthology.orchestra.run.smithi031.stdout:34359738370 2023-08-26T13:31:28.735 INFO:tasks.ceph.ceph_manager.cluster1:need seq 34359738371 got 34359738370 for osd.2 2023-08-26T13:31:28.747 INFO:tasks.ceph.ceph_manager.cluster1:need seq 34359738371 got 34359738370 for osd.0 2023-08-26T13:31:29.725 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 osd last-stat-seq osd.1 2023-08-26T13:31:29.736 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 osd last-stat-seq osd.2 2023-08-26T13:31:29.748 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 osd last-stat-seq osd.0 2023-08-26T13:31:30.104 INFO:teuthology.orchestra.run.smithi031.stdout:34359738372 2023-08-26T13:31:30.119 INFO:tasks.ceph.ceph_manager.cluster1:need seq 34359738373 got 34359738372 for osd.1 2023-08-26T13:31:30.127 INFO:teuthology.orchestra.run.smithi031.stdout:34359738370 2023-08-26T13:31:30.141 INFO:tasks.ceph.ceph_manager.cluster1:need seq 34359738371 got 34359738370 for osd.2 2023-08-26T13:31:30.143 INFO:teuthology.orchestra.run.smithi031.stdout:34359738370 2023-08-26T13:31:30.157 INFO:tasks.ceph.ceph_manager.cluster1:need seq 34359738371 got 34359738370 for osd.0 2023-08-26T13:31:31.120 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 osd last-stat-seq osd.1 2023-08-26T13:31:31.143 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 osd last-stat-seq osd.2 2023-08-26T13:31:31.158 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 osd last-stat-seq osd.0 2023-08-26T13:31:31.483 INFO:teuthology.orchestra.run.smithi031.stdout:34359738373 2023-08-26T13:31:31.497 INFO:tasks.ceph.ceph_manager.cluster1:need seq 34359738373 got 34359738373 for osd.1 2023-08-26T13:31:31.498 DEBUG:teuthology.parallel:result is None 2023-08-26T13:31:31.540 INFO:teuthology.orchestra.run.smithi031.stdout:34359738371 2023-08-26T13:31:31.547 INFO:teuthology.orchestra.run.smithi031.stdout:34359738371 2023-08-26T13:31:31.556 INFO:tasks.ceph.ceph_manager.cluster1:need seq 34359738371 got 34359738371 for osd.0 2023-08-26T13:31:31.557 DEBUG:teuthology.parallel:result is None 2023-08-26T13:31:31.563 INFO:tasks.ceph.ceph_manager.cluster1:need seq 34359738371 got 34359738371 for osd.2 2023-08-26T13:31:31.563 DEBUG:teuthology.parallel:result is None 2023-08-26T13:31:31.564 INFO:tasks.ceph.ceph_manager.cluster1:waiting for clean 2023-08-26T13:31:31.564 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 pg dump --format=json 2023-08-26T13:31:31.933 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-08-26T13:31:31.936 INFO:teuthology.orchestra.run.smithi031.stderr:dumped all 2023-08-26T13:31:31.947 INFO:teuthology.orchestra.run.smithi031.stdout:{"pg_ready":true,"pg_map":{"version":16,"stamp":"2023-08-26T13:31:30.041375+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":9,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":82572,"kb_used_data":1504,"kb_used_omap":5,"kb_used_meta":81018,"kb_avail":283032948,"statfs":{"total":289910292480,"available":289825738752,"internally_reserved":0,"allocated":1540096,"data_stored":1278171,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":5343,"internal_metadata":82963233},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":10,"apply_latency_ms":10,"commit_latency_ns":10000000,"apply_latency_ns":10000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.047543"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":9,"reported_epoch":13,"state":"active+clean","last_fresh":"2023-08-26T13:31:27.011966+0000","last_change":"2023-08-26T13:31:27.005178+0000","last_active":"2023-08-26T13:31:27.011966+0000","last_peered":"2023-08-26T13:31:27.011966+0000","last_clean":"2023-08-26T13:31:27.011966+0000","last_became_active":"2023-08-26T13:31:27.004840+0000","last_became_peered":"2023-08-26T13:31:27.004840+0000","last_unstale":"2023-08-26T13:31:27.011966+0000","last_undegraded":"2023-08-26T13:31:27.011966+0000","last_fullsized":"2023-08-26T13:31:27.011966+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-26T13:31:25.989878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-26T13:31:25.989878+0000","last_clean_scrub_stamp":"2023-08-26T13:31:25.989878+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-27T20:38:08.005218+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":9,"reported_epoch":13,"state":"active+clean","last_fresh":"2023-08-26T13:31:27.011909+0000","last_change":"2023-08-26T13:31:27.005155+0000","last_active":"2023-08-26T13:31:27.011909+0000","last_peered":"2023-08-26T13:31:27.011909+0000","last_clean":"2023-08-26T13:31:27.011909+0000","last_became_active":"2023-08-26T13:31:27.004881+0000","last_became_peered":"2023-08-26T13:31:27.004881+0000","last_unstale":"2023-08-26T13:31:27.011909+0000","last_undegraded":"2023-08-26T13:31:27.011909+0000","last_fullsized":"2023-08-26T13:31:27.011909+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-26T13:31:25.989878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-26T13:31:25.989878+0000","last_clean_scrub_stamp":"2023-08-26T13:31:25.989878+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-27T22:35:27.587497+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":9,"reported_epoch":13,"state":"active+clean","last_fresh":"2023-08-26T13:31:27.012128+0000","last_change":"2023-08-26T13:31:27.008475+0000","last_active":"2023-08-26T13:31:27.012128+0000","last_peered":"2023-08-26T13:31:27.012128+0000","last_clean":"2023-08-26T13:31:27.012128+0000","last_became_active":"2023-08-26T13:31:27.008118+0000","last_became_peered":"2023-08-26T13:31:27.008118+0000","last_unstale":"2023-08-26T13:31:27.012128+0000","last_undegraded":"2023-08-26T13:31:27.012128+0000","last_fullsized":"2023-08-26T13:31:27.012128+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-26T13:31:25.989878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-26T13:31:25.989878+0000","last_clean_scrub_stamp":"2023-08-26T13:31:25.989878+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-27T21:18:57.514664+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":9,"reported_epoch":13,"state":"active+clean","last_fresh":"2023-08-26T13:31:27.009211+0000","last_change":"2023-08-26T13:31:27.006130+0000","last_active":"2023-08-26T13:31:27.009211+0000","last_peered":"2023-08-26T13:31:27.009211+0000","last_clean":"2023-08-26T13:31:27.009211+0000","last_became_active":"2023-08-26T13:31:27.005887+0000","last_became_peered":"2023-08-26T13:31:27.005887+0000","last_unstale":"2023-08-26T13:31:27.009211+0000","last_undegraded":"2023-08-26T13:31:27.009211+0000","last_fullsized":"2023-08-26T13:31:27.009211+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-26T13:31:25.989878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-26T13:31:25.989878+0000","last_clean_scrub_stamp":"2023-08-26T13:31:25.989878+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-27T17:23:45.299049+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"10'97","reported_seq":142,"reported_epoch":13,"state":"active+clean","last_fresh":"2023-08-26T13:31:26.999186+0000","last_change":"2023-08-26T13:31:22.014049+0000","last_active":"2023-08-26T13:31:26.999186+0000","last_peered":"2023-08-26T13:31:26.999186+0000","last_clean":"2023-08-26T13:31:26.999186+0000","last_became_active":"2023-08-26T13:31:22.012989+0000","last_became_peered":"2023-08-26T13:31:22.012989+0000","last_unstale":"2023-08-26T13:31:26.999186+0000","last_undegraded":"2023-08-26T13:31:26.999186+0000","last_fullsized":"2023-08-26T13:31:26.999186+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-26T13:31:20.981385+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-26T13:31:20.981385+0000","last_clean_scrub_stamp":"2023-08-26T13:31:20.981385+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-28T01:09:44.217004+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":9,"reported_epoch":13,"state":"active+clean","last_fresh":"2023-08-26T13:31:27.011876+0000","last_change":"2023-08-26T13:31:27.004458+0000","last_active":"2023-08-26T13:31:27.011876+0000","last_peered":"2023-08-26T13:31:27.011876+0000","last_clean":"2023-08-26T13:31:27.011876+0000","last_became_active":"2023-08-26T13:31:27.004107+0000","last_became_peered":"2023-08-26T13:31:27.004107+0000","last_unstale":"2023-08-26T13:31:27.011876+0000","last_undegraded":"2023-08-26T13:31:27.011876+0000","last_fullsized":"2023-08-26T13:31:27.011876+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-26T13:31:25.989878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-26T13:31:25.989878+0000","last_clean_scrub_stamp":"2023-08-26T13:31:25.989878+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-27T19:55:30.457713+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":9,"reported_epoch":13,"state":"active+clean","last_fresh":"2023-08-26T13:31:27.009206+0000","last_change":"2023-08-26T13:31:27.006034+0000","last_active":"2023-08-26T13:31:27.009206+0000","last_peered":"2023-08-26T13:31:27.009206+0000","last_clean":"2023-08-26T13:31:27.009206+0000","last_became_active":"2023-08-26T13:31:27.005729+0000","last_became_peered":"2023-08-26T13:31:27.005729+0000","last_unstale":"2023-08-26T13:31:27.009206+0000","last_undegraded":"2023-08-26T13:31:27.009206+0000","last_fullsized":"2023-08-26T13:31:27.009206+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-26T13:31:25.989878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-26T13:31:25.989878+0000","last_clean_scrub_stamp":"2023-08-26T13:31:25.989878+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-27T14:21:39.568605+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":11,"reported_epoch":14,"state":"active+clean","last_fresh":"2023-08-26T13:31:28.003679+0000","last_change":"2023-08-26T13:31:27.009922+0000","last_active":"2023-08-26T13:31:28.003679+0000","last_peered":"2023-08-26T13:31:28.003679+0000","last_clean":"2023-08-26T13:31:28.003679+0000","last_became_active":"2023-08-26T13:31:27.009616+0000","last_became_peered":"2023-08-26T13:31:27.009616+0000","last_unstale":"2023-08-26T13:31:28.003679+0000","last_undegraded":"2023-08-26T13:31:28.003679+0000","last_fullsized":"2023-08-26T13:31:28.003679+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-26T13:31:25.989878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-26T13:31:25.989878+0000","last_clean_scrub_stamp":"2023-08-26T13:31:25.989878+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-27T17:26:00.961520+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":9,"reported_epoch":13,"state":"active+clean","last_fresh":"2023-08-26T13:31:27.012081+0000","last_change":"2023-08-26T13:31:27.008458+0000","last_active":"2023-08-26T13:31:27.012081+0000","last_peered":"2023-08-26T13:31:27.012081+0000","last_clean":"2023-08-26T13:31:27.012081+0000","last_became_active":"2023-08-26T13:31:27.008131+0000","last_became_peered":"2023-08-26T13:31:27.008131+0000","last_unstale":"2023-08-26T13:31:27.012081+0000","last_undegraded":"2023-08-26T13:31:27.012081+0000","last_fullsized":"2023-08-26T13:31:27.012081+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-26T13:31:25.989878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-26T13:31:25.989878+0000","last_clean_scrub_stamp":"2023-08-26T13:31:25.989878+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-27T21:11:30.757281+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738371,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27136,"kb_used_data":112,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344704,"statfs":{"total":96636764160,"available":96608976896,"internally_reserved":0,"allocated":114688,"data_stored":31319,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1781,"internal_metadata":27654411},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.39300000000000002}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.45400000000000001}]}]},{"osd":0,"up_from":8,"seq":34359738371,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27724,"kb_used_data":700,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344116,"statfs":{"total":96636764160,"available":96608374784,"internally_reserved":0,"allocated":716800,"data_stored":625165,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1781,"internal_metadata":27654411},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.34499999999999997}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.253}]}]},{"osd":1,"up_from":8,"seq":34359738373,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27712,"kb_used_data":692,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344128,"statfs":{"total":96636764160,"available":96608387072,"internally_reserved":0,"allocated":708608,"data_stored":621687,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1781,"internal_metadata":27654411},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.28699999999999998}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.434}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-08-26T13:31:31.948 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 pg dump --format=json 2023-08-26T13:31:32.280 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-08-26T13:31:32.283 INFO:teuthology.orchestra.run.smithi031.stderr:dumped all 2023-08-26T13:31:32.294 INFO:teuthology.orchestra.run.smithi031.stdout:{"pg_ready":true,"pg_map":{"version":17,"stamp":"2023-08-26T13:31:32.041674+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":17,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":82572,"kb_used_data":1504,"kb_used_omap":5,"kb_used_meta":81018,"kb_avail":283032948,"statfs":{"total":289910292480,"available":289825738752,"internally_reserved":0,"allocated":1540096,"data_stored":1278171,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":5343,"internal_metadata":82963233},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.047843"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":11,"reported_epoch":14,"state":"active+clean","last_fresh":"2023-08-26T13:31:29.643664+0000","last_change":"2023-08-26T13:31:27.005178+0000","last_active":"2023-08-26T13:31:29.643664+0000","last_peered":"2023-08-26T13:31:29.643664+0000","last_clean":"2023-08-26T13:31:29.643664+0000","last_became_active":"2023-08-26T13:31:27.004840+0000","last_became_peered":"2023-08-26T13:31:27.004840+0000","last_unstale":"2023-08-26T13:31:29.643664+0000","last_undegraded":"2023-08-26T13:31:29.643664+0000","last_fullsized":"2023-08-26T13:31:29.643664+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-26T13:31:25.989878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-26T13:31:25.989878+0000","last_clean_scrub_stamp":"2023-08-26T13:31:25.989878+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-27T23:23:49.291755+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":11,"reported_epoch":14,"state":"active+clean","last_fresh":"2023-08-26T13:31:29.643619+0000","last_change":"2023-08-26T13:31:27.005155+0000","last_active":"2023-08-26T13:31:29.643619+0000","last_peered":"2023-08-26T13:31:29.643619+0000","last_clean":"2023-08-26T13:31:29.643619+0000","last_became_active":"2023-08-26T13:31:27.004881+0000","last_became_peered":"2023-08-26T13:31:27.004881+0000","last_unstale":"2023-08-26T13:31:29.643619+0000","last_undegraded":"2023-08-26T13:31:29.643619+0000","last_fullsized":"2023-08-26T13:31:29.643619+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-26T13:31:25.989878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-26T13:31:25.989878+0000","last_clean_scrub_stamp":"2023-08-26T13:31:25.989878+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-27T15:06:00.597524+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":11,"reported_epoch":14,"state":"active+clean","last_fresh":"2023-08-26T13:31:29.643519+0000","last_change":"2023-08-26T13:31:27.008475+0000","last_active":"2023-08-26T13:31:29.643519+0000","last_peered":"2023-08-26T13:31:29.643519+0000","last_clean":"2023-08-26T13:31:29.643519+0000","last_became_active":"2023-08-26T13:31:27.008118+0000","last_became_peered":"2023-08-26T13:31:27.008118+0000","last_unstale":"2023-08-26T13:31:29.643519+0000","last_undegraded":"2023-08-26T13:31:29.643519+0000","last_fullsized":"2023-08-26T13:31:29.643519+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-26T13:31:25.989878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-26T13:31:25.989878+0000","last_clean_scrub_stamp":"2023-08-26T13:31:25.989878+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-27T17:07:57.606904+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":9,"reported_epoch":13,"state":"active+clean","last_fresh":"2023-08-26T13:31:27.009211+0000","last_change":"2023-08-26T13:31:27.006130+0000","last_active":"2023-08-26T13:31:27.009211+0000","last_peered":"2023-08-26T13:31:27.009211+0000","last_clean":"2023-08-26T13:31:27.009211+0000","last_became_active":"2023-08-26T13:31:27.005887+0000","last_became_peered":"2023-08-26T13:31:27.005887+0000","last_unstale":"2023-08-26T13:31:27.009211+0000","last_undegraded":"2023-08-26T13:31:27.009211+0000","last_fullsized":"2023-08-26T13:31:27.009211+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-26T13:31:25.989878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-26T13:31:25.989878+0000","last_clean_scrub_stamp":"2023-08-26T13:31:25.989878+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-27T17:23:45.299049+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"10'97","reported_seq":144,"reported_epoch":14,"state":"active+clean","last_fresh":"2023-08-26T13:31:29.643480+0000","last_change":"2023-08-26T13:31:22.014049+0000","last_active":"2023-08-26T13:31:29.643480+0000","last_peered":"2023-08-26T13:31:29.643480+0000","last_clean":"2023-08-26T13:31:29.643480+0000","last_became_active":"2023-08-26T13:31:22.012989+0000","last_became_peered":"2023-08-26T13:31:22.012989+0000","last_unstale":"2023-08-26T13:31:29.643480+0000","last_undegraded":"2023-08-26T13:31:29.643480+0000","last_fullsized":"2023-08-26T13:31:29.643480+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-26T13:31:20.981385+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-26T13:31:20.981385+0000","last_clean_scrub_stamp":"2023-08-26T13:31:20.981385+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-28T01:00:01.891617+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":11,"reported_epoch":14,"state":"active+clean","last_fresh":"2023-08-26T13:31:29.643542+0000","last_change":"2023-08-26T13:31:27.004458+0000","last_active":"2023-08-26T13:31:29.643542+0000","last_peered":"2023-08-26T13:31:29.643542+0000","last_clean":"2023-08-26T13:31:29.643542+0000","last_became_active":"2023-08-26T13:31:27.004107+0000","last_became_peered":"2023-08-26T13:31:27.004107+0000","last_unstale":"2023-08-26T13:31:29.643542+0000","last_undegraded":"2023-08-26T13:31:29.643542+0000","last_fullsized":"2023-08-26T13:31:29.643542+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-26T13:31:25.989878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-26T13:31:25.989878+0000","last_clean_scrub_stamp":"2023-08-26T13:31:25.989878+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-27T20:31:59.874303+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":9,"reported_epoch":13,"state":"active+clean","last_fresh":"2023-08-26T13:31:27.009206+0000","last_change":"2023-08-26T13:31:27.006034+0000","last_active":"2023-08-26T13:31:27.009206+0000","last_peered":"2023-08-26T13:31:27.009206+0000","last_clean":"2023-08-26T13:31:27.009206+0000","last_became_active":"2023-08-26T13:31:27.005729+0000","last_became_peered":"2023-08-26T13:31:27.005729+0000","last_unstale":"2023-08-26T13:31:27.009206+0000","last_undegraded":"2023-08-26T13:31:27.009206+0000","last_fullsized":"2023-08-26T13:31:27.009206+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-26T13:31:25.989878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-26T13:31:25.989878+0000","last_clean_scrub_stamp":"2023-08-26T13:31:25.989878+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-27T14:21:39.568605+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":11,"reported_epoch":14,"state":"active+clean","last_fresh":"2023-08-26T13:31:28.003679+0000","last_change":"2023-08-26T13:31:27.009922+0000","last_active":"2023-08-26T13:31:28.003679+0000","last_peered":"2023-08-26T13:31:28.003679+0000","last_clean":"2023-08-26T13:31:28.003679+0000","last_became_active":"2023-08-26T13:31:27.009616+0000","last_became_peered":"2023-08-26T13:31:27.009616+0000","last_unstale":"2023-08-26T13:31:28.003679+0000","last_undegraded":"2023-08-26T13:31:28.003679+0000","last_fullsized":"2023-08-26T13:31:28.003679+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-26T13:31:25.989878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-26T13:31:25.989878+0000","last_clean_scrub_stamp":"2023-08-26T13:31:25.989878+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-27T17:26:00.961520+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":11,"reported_epoch":14,"state":"active+clean","last_fresh":"2023-08-26T13:31:29.643501+0000","last_change":"2023-08-26T13:31:27.008458+0000","last_active":"2023-08-26T13:31:29.643501+0000","last_peered":"2023-08-26T13:31:29.643501+0000","last_clean":"2023-08-26T13:31:29.643501+0000","last_became_active":"2023-08-26T13:31:27.008131+0000","last_became_peered":"2023-08-26T13:31:27.008131+0000","last_unstale":"2023-08-26T13:31:29.643501+0000","last_undegraded":"2023-08-26T13:31:29.643501+0000","last_fullsized":"2023-08-26T13:31:29.643501+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-26T13:31:25.989878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-26T13:31:25.989878+0000","last_clean_scrub_stamp":"2023-08-26T13:31:25.989878+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-27T20:33:59.508215+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738372,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27136,"kb_used_data":112,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344704,"statfs":{"total":96636764160,"available":96608976896,"internally_reserved":0,"allocated":114688,"data_stored":31319,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1781,"internal_metadata":27654411},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.39300000000000002}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.45400000000000001}]}]},{"osd":0,"up_from":8,"seq":34359738372,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27724,"kb_used_data":700,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344116,"statfs":{"total":96636764160,"available":96608374784,"internally_reserved":0,"allocated":716800,"data_stored":625165,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1781,"internal_metadata":27654411},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.34499999999999997}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.253}]}]},{"osd":1,"up_from":8,"seq":34359738374,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27712,"kb_used_data":692,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344128,"statfs":{"total":96636764160,"available":96608387072,"internally_reserved":0,"allocated":708608,"data_stored":621687,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1781,"internal_metadata":27654411},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.49199999999999999}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.46800000000000003}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-08-26T13:31:32.295 INFO:tasks.ceph.ceph_manager.cluster1:clean! 2023-08-26T13:31:32.295 INFO:tasks.ceph:Waiting until ceph cluster cluster1 is healthy... 2023-08-26T13:31:32.296 INFO:tasks.ceph.ceph_manager.cluster1:wait_until_healthy 2023-08-26T13:31:32.296 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 health --format=json 2023-08-26T13:31:32.670 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-08-26T13:31:32.671 INFO:teuthology.orchestra.run.smithi031.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2023-08-26T13:31:32.684 INFO:tasks.ceph.ceph_manager.cluster1:wait_until_healthy done 2023-08-26T13:31:32.684 INFO:teuthology.run_tasks:Running task ceph... 2023-08-26T13:31:32.698 INFO:tasks.ceph:Creating ceph cluster cluster2... 2023-08-26T13:31:32.698 INFO:tasks.ceph:config {'conf': {'client': {'debug rbd': 20, 'debug rbd_mirror': 15, 'log to stderr': False, 'rbd default features': 125, 'rbd mirror image policy type': 'simple'}, 'client.mirror.0': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.1': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.2': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.3': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'global': {'mon client directed command retry': 5, 'ms inject socket failures': 5000}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20}, 'osd': {'bdev async discard': True, 'bdev enable discard': True, 'bluestore allocator': 'stupid', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore'}}, 'fs': 'xfs', 'mkfs_options': None, 'mount_options': None, 'skip_mgr_daemons': False, 'log_ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(OSD_SLOW_PING_TIME'], 'cpu_profile': set(), 'cluster': 'cluster2', 'mon_bind_msgr2': True, 'mon_bind_addrvec': True} 2023-08-26T13:31:32.699 INFO:tasks.ceph:ctx.config {'archive_path': '/home/teuthworker/archive/yuriw-2023-08-25_17:40:36-rbd-main-distro-default-smithi/7380431', 'branch': 'main', 'description': 'rbd/mirror-thrash/{base/install clients/mirror cluster/{2-node openstack} msgr-failures/few objectstore/bluestore-stupid policy/simple rbd-mirror/four-per-cluster supported-random-distro$/{rhel_8} workloads/rbd-mirror-snapshot-stress-workunit-exclusive-lock}', 'email': 'yweinste@redhat.com', 'first_in_suite': False, 'job_id': '7380431', 'kernel': {'kdb': True, 'sha1': 'distro'}, 'ktype': 'distro', 'last_in_suite': False, 'machine_type': 'smithi', 'meta': [{'desc': 'run two ceph clusters and install rbd-mirror'}, {'desc': 'configure the permissions for client.mirror'}, {'desc': '2 ceph clusters with 1 mon and 3 osds each'}, {'desc': 'run four rbd-mirror daemons per cluster'}, {'desc': 'run the rbd_mirror_stress.sh workunit to test the rbd-mirror daemon'}], 'name': 'yuriw-2023-08-25_17:40:36-rbd-main-distro-default-smithi', 'no_nested_subset': False, 'nuke-on-error': True, 'openstack': [{'volumes': {'count': 3, 'size': 30}}], 'os_type': 'rhel', 'os_version': '8.6', 'overrides': {'admin_socket': {'branch': 'main'}, 'ceph': {'conf': {'client': {'debug rbd': 20, 'debug rbd_mirror': 15, 'log to stderr': False, 'rbd default features': 125, 'rbd mirror image policy type': 'simple'}, 'client.mirror.0': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.1': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.2': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.3': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'global': {'mon client directed command retry': 5, 'ms inject socket failures': 5000}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20}, 'osd': {'bdev async discard': True, 'bdev enable discard': True, 'bluestore allocator': 'stupid', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore'}}, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(OSD_SLOW_PING_TIME'], 'sha1': 'e7eb9853fddaf661c68f207346131a7e00d8ed2f'}, 'ceph-deploy': {'bluestore': True, 'conf': {'client': {'log file': '/var/log/ceph/ceph-$name.$pid.log'}, 'mon': {}, 'osd': {'bdev async discard': True, 'bdev enable discard': True, 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore'}}, 'fs': 'xfs'}, 'install': {'ceph': {'flavor': 'default', 'sha1': 'e7eb9853fddaf661c68f207346131a7e00d8ed2f'}}, 'selinux': {'whitelist': ['scontext=system_u:system_r:logrotate_t:s0']}, 'thrashosds': {'bdev_inject_crash': 2, 'bdev_inject_crash_probability': 0.5}, 'workunit': {'branch': 'main', 'sha1': 'e7eb9853fddaf661c68f207346131a7e00d8ed2f'}}, 'owner': 'scheduled_yuriw@teuthology', 'priority': 75, 'repo': 'https://github.com/ceph/ceph-ci.git', 'roles': [['cluster1.mon.a', 'cluster1.mgr.x', 'cluster2.mgr.x', 'cluster1.osd.0', 'cluster1.osd.1', 'cluster1.osd.2', 'cluster1.client.0', 'cluster2.client.0'], ['cluster2.mon.a', 'cluster2.osd.0', 'cluster2.osd.1', 'cluster2.osd.2', 'cluster1.client.mirror', 'cluster1.client.mirror.0', 'cluster1.client.mirror.1', 'cluster1.client.mirror.2', 'cluster1.client.mirror.3', 'cluster1.client.mirror.4', 'cluster1.client.mirror.5', 'cluster1.client.mirror.6', 'cluster2.client.mirror', 'cluster2.client.mirror.0', 'cluster2.client.mirror.1', 'cluster2.client.mirror.2', 'cluster2.client.mirror.3', 'cluster2.client.mirror.4', 'cluster2.client.mirror.5', 'cluster2.client.mirror.6']], 'seed': 5715, 'sha1': 'e7eb9853fddaf661c68f207346131a7e00d8ed2f', 'sleep_before_teardown': 0, 'subset': '111/120000', 'suite': 'rbd', 'suite_branch': 'main', 'suite_path': '/home/teuthworker/src/github.com_ceph_ceph-c_e7eb9853fddaf661c68f207346131a7e00d8ed2f/qa', 'suite_relpath': 'qa', 'suite_repo': 'https://github.com/ceph/ceph-ci.git', 'suite_sha1': 'e7eb9853fddaf661c68f207346131a7e00d8ed2f', 'targets': {'smithi031.front.sepia.ceph.com': 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCpUxo5ay/ucdkX9rleNMm8Ofp24XRXUtJPOK3Ji3XBpTiUgmmwneaY6316+3CG+k4K0p3IebanuyTvZApjm02/dNCvu8yD3rjEpWVgIuqYH8O3lEdN9DXcCX+7xHJE0m18+ZPsJQVsNrNJxY+5zqyNnBc8jpZq2LStsWwPOKPWhauANRkOBFSGDoLppWZbLx/yJs+R0qoNLLK9LpKQTUnqsapXvbD27OpPahzj/qbNxFPPyY6XrrldZ6qEO+EjJOujFLH3gzv0kJu10DCuzxrpz6ston4RPi1O3GgcHnF4Mqr/W9VwZ94UTMh+k4dIFRVqJEKrcwRHa5kFsvH/WhVx0Ygr/MiV/Q86mmV7JT+/MpzIRTtWyei5L9IBeZIE9cwzj+mOk6SD/t1roIVEGVV/uF2AyFDVBPr6F8oZ822OGpUyyaDsWTitPNdTX4TC3zdSrLc9tyDa1P74eH/ZLeTSf+C6pM/cepOhQaa6XzpDiPSuJUm9HOQxUlEAvcqa7Ac=', 'smithi124.front.sepia.ceph.com': 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCpUxo5ay/ucdkX9rleNMm8Ofp24XRXUtJPOK3Ji3XBpTiUgmmwneaY6316+3CG+k4K0p3IebanuyTvZApjm02/dNCvu8yD3rjEpWVgIuqYH8O3lEdN9DXcCX+7xHJE0m18+ZPsJQVsNrNJxY+5zqyNnBc8jpZq2LStsWwPOKPWhauANRkOBFSGDoLppWZbLx/yJs+R0qoNLLK9LpKQTUnqsapXvbD27OpPahzj/qbNxFPPyY6XrrldZ6qEO+EjJOujFLH3gzv0kJu10DCuzxrpz6ston4RPi1O3GgcHnF4Mqr/W9VwZ94UTMh+k4dIFRVqJEKrcwRHa5kFsvH/WhVx0Ygr/MiV/Q86mmV7JT+/MpzIRTtWyei5L9IBeZIE9cwzj+mOk6SD/t1roIVEGVV/uF2AyFDVBPr6F8oZ822OGpUyyaDsWTitPNdTX4TC3zdSrLc9tyDa1P74eH/ZLeTSf+C6pM/cepOhQaa6XzpDiPSuJUm9HOQxUlEAvcqa7Ac='}, 'tasks': [{'internal.check_packages': None}, {'internal.buildpackages_prep': None}, {'internal.save_config': None}, {'internal.check_lock': None}, {'internal.add_remotes': None}, {'console_log': None}, {'internal.connect': None}, {'internal.push_inventory': None}, {'internal.serialize_remote_roles': None}, {'internal.check_conflict': None}, {'internal.check_ceph_data': None}, {'internal.vm_setup': None}, {'kernel': {'kdb': True, 'sha1': 'distro'}}, {'internal.base': None}, {'internal.archive_upload': None}, {'internal.archive': None}, {'internal.coredump': None}, {'internal.sudo': None}, {'internal.syslog': None}, {'internal.timer': None}, {'pcp': None}, {'selinux': None}, {'ansible.cephlab': None}, {'clock': None}, {'install': {'extra_packages': ['rbd-mirror'], 'flavor': 'default', 'sha1': 'e7eb9853fddaf661c68f207346131a7e00d8ed2f'}}, {'ceph': {'cluster': 'cluster1', 'conf': {'client': {'debug rbd': 20, 'debug rbd_mirror': 15, 'log to stderr': False, 'rbd default features': 125, 'rbd mirror image policy type': 'simple'}, 'client.mirror.0': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.1': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.2': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.3': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'global': {'mon client directed command retry': 5, 'ms inject socket failures': 5000}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20}, 'osd': {'bdev async discard': True, 'bdev enable discard': True, 'bluestore allocator': 'stupid', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore'}}, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(OSD_SLOW_PING_TIME'], 'sha1': 'e7eb9853fddaf661c68f207346131a7e00d8ed2f'}}, {'ceph': {'cluster': 'cluster2', 'conf': {'client': {'debug rbd': 20, 'debug rbd_mirror': 15, 'log to stderr': False, 'rbd default features': 125, 'rbd mirror image policy type': 'simple'}, 'client.mirror.0': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.1': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.2': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.3': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'global': {'mon client directed command retry': 5, 'ms inject socket failures': 5000}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20}, 'osd': {'bdev async discard': True, 'bdev enable discard': True, 'bluestore allocator': 'stupid', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore'}}, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(OSD_SLOW_PING_TIME'], 'sha1': 'e7eb9853fddaf661c68f207346131a7e00d8ed2f'}}, {'exec': {'cluster1.client.mirror.0': ["sudo ceph --cluster cluster1 auth caps client.mirror mon 'profile rbd-mirror-peer' osd 'profile rbd'", "sudo ceph --cluster cluster1 auth caps client.mirror.0 mon 'profile rbd-mirror' osd 'profile rbd'", "sudo ceph --cluster cluster1 auth caps client.mirror.1 mon 'profile rbd-mirror' osd 'profile rbd'", "sudo ceph --cluster cluster1 auth caps client.mirror.2 mon 'profile rbd-mirror' osd 'profile rbd'", "sudo ceph --cluster cluster1 auth caps client.mirror.3 mon 'profile rbd-mirror' osd 'profile rbd'", "sudo ceph --cluster cluster2 auth caps client.mirror mon 'profile rbd-mirror-peer' osd 'profile rbd'", "sudo ceph --cluster cluster2 auth caps client.mirror.0 mon 'profile rbd-mirror' osd 'profile rbd'", "sudo ceph --cluster cluster2 auth caps client.mirror.1 mon 'profile rbd-mirror' osd 'profile rbd'", "sudo ceph --cluster cluster2 auth caps client.mirror.2 mon 'profile rbd-mirror' osd 'profile rbd'", "sudo ceph --cluster cluster2 auth caps client.mirror.3 mon 'profile rbd-mirror' osd 'profile rbd'"]}}, {'rbd-mirror': {'client': 'cluster1.client.mirror.0', 'thrash': True}}, {'rbd-mirror': {'client': 'cluster1.client.mirror.1', 'thrash': True}}, {'rbd-mirror': {'client': 'cluster1.client.mirror.2', 'thrash': True}}, {'rbd-mirror': {'client': 'cluster1.client.mirror.3', 'thrash': True}}, {'rbd-mirror': {'client': 'cluster2.client.mirror.0', 'thrash': True}}, {'rbd-mirror': {'client': 'cluster2.client.mirror.1', 'thrash': True}}, {'rbd-mirror': {'client': 'cluster2.client.mirror.2', 'thrash': True}}, {'rbd-mirror': {'client': 'cluster2.client.mirror.3', 'thrash': True}}, {'rbd-mirror-thrash': {'cluster': 'cluster1'}}, {'rbd-mirror-thrash': {'cluster': 'cluster2'}}, {'workunit': {'clients': {'cluster1.client.mirror': ['rbd/rbd_mirror_stress.sh']}, 'env': {'CEPH_ARGS': '', 'MIRROR_IMAGE_MODE': 'snapshot', 'MIRROR_POOL_MODE': 'image', 'RBD_IMAGE_FEATURES': 'layering,exclusive-lock', 'RBD_MIRROR_INSTANCES': '4', 'RBD_MIRROR_USE_EXISTING_CLUSTER': '1', 'RBD_MIRROR_USE_RBD_MIRROR': '1'}, 'timeout': '6h'}}], 'teuthology': {'fragments_dropped': [], 'meta': {}, 'postmerge': []}, 'teuthology_branch': 'main', 'teuthology_sha1': '449a1bc2027504e7b3c3d7b30fa4178906581da7', 'timestamp': '2023-08-25_17:40:36', 'tube': 'smithi', 'user': 'yuriw', 'verbose': True, 'worker_log': '/home/teuthworker/archive/worker_logs/dispatcher.smithi.2002'} 2023-08-26T13:31:32.700 DEBUG:teuthology.orchestra.run.smithi031:> install -d -m0755 -- /home/ubuntu/cephtest/cluster2.data 2023-08-26T13:31:32.702 DEBUG:teuthology.orchestra.run.smithi124:> install -d -m0755 -- /home/ubuntu/cephtest/cluster2.data 2023-08-26T13:31:32.722 DEBUG:teuthology.orchestra.run.smithi031:> sudo install -d -m0777 -- /var/run/ceph 2023-08-26T13:31:32.759 DEBUG:teuthology.orchestra.run.smithi124:> sudo install -d -m0777 -- /var/run/ceph 2023-08-26T13:31:32.794 DEBUG:teuthology.orchestra.run.smithi124:> set -ex 2023-08-26T13:31:32.795 DEBUG:teuthology.orchestra.run.smithi124:> dd if=/scratch_devs of=/dev/stdout 2023-08-26T13:31:32.855 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2023-08-26T13:31:32.855 DEBUG:teuthology.orchestra.run.smithi124:> stat /dev/vg_nvme/lv_1 2023-08-26T13:31:32.915 INFO:teuthology.orchestra.run.smithi124.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2023-08-26T13:31:32.916 INFO:teuthology.orchestra.run.smithi124.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-08-26T13:31:32.916 INFO:teuthology.orchestra.run.smithi124.stdout:Device: 6h/6d Inode: 390934 Links: 1 2023-08-26T13:31:32.916 INFO:teuthology.orchestra.run.smithi124.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-08-26T13:31:32.916 INFO:teuthology.orchestra.run.smithi124.stdout:Context: system_u:object_r:device_t:s0 2023-08-26T13:31:32.916 INFO:teuthology.orchestra.run.smithi124.stdout:Access: 2023-08-26 13:30:30.231128922 +0000 2023-08-26T13:31:32.917 INFO:teuthology.orchestra.run.smithi124.stdout:Modify: 2023-08-26 13:24:29.040851955 +0000 2023-08-26T13:31:32.917 INFO:teuthology.orchestra.run.smithi124.stdout:Change: 2023-08-26 13:24:29.040851955 +0000 2023-08-26T13:31:32.917 INFO:teuthology.orchestra.run.smithi124.stdout: Birth: - 2023-08-26T13:31:32.917 DEBUG:teuthology.orchestra.run.smithi124:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2023-08-26T13:31:32.986 INFO:teuthology.orchestra.run.smithi124.stderr:1+0 records in 2023-08-26T13:31:32.986 INFO:teuthology.orchestra.run.smithi124.stderr:1+0 records out 2023-08-26T13:31:32.986 INFO:teuthology.orchestra.run.smithi124.stderr:512 bytes copied, 0.000268796 s, 1.9 MB/s 2023-08-26T13:31:32.987 DEBUG:teuthology.orchestra.run.smithi124:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2023-08-26T13:31:33.049 DEBUG:teuthology.orchestra.run.smithi124:> stat /dev/vg_nvme/lv_2 2023-08-26T13:31:33.109 INFO:teuthology.orchestra.run.smithi124.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2023-08-26T13:31:33.109 INFO:teuthology.orchestra.run.smithi124.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-08-26T13:31:33.109 INFO:teuthology.orchestra.run.smithi124.stdout:Device: 6h/6d Inode: 391002 Links: 1 2023-08-26T13:31:33.109 INFO:teuthology.orchestra.run.smithi124.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-08-26T13:31:33.110 INFO:teuthology.orchestra.run.smithi124.stdout:Context: system_u:object_r:device_t:s0 2023-08-26T13:31:33.110 INFO:teuthology.orchestra.run.smithi124.stdout:Access: 2023-08-26 13:30:30.231128922 +0000 2023-08-26T13:31:33.110 INFO:teuthology.orchestra.run.smithi124.stdout:Modify: 2023-08-26 13:24:29.537843428 +0000 2023-08-26T13:31:33.110 INFO:teuthology.orchestra.run.smithi124.stdout:Change: 2023-08-26 13:24:29.537843428 +0000 2023-08-26T13:31:33.110 INFO:teuthology.orchestra.run.smithi124.stdout: Birth: - 2023-08-26T13:31:33.111 DEBUG:teuthology.orchestra.run.smithi124:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2023-08-26T13:31:33.179 INFO:teuthology.orchestra.run.smithi124.stderr:1+0 records in 2023-08-26T13:31:33.180 INFO:teuthology.orchestra.run.smithi124.stderr:1+0 records out 2023-08-26T13:31:33.180 INFO:teuthology.orchestra.run.smithi124.stderr:512 bytes copied, 0.000282441 s, 1.8 MB/s 2023-08-26T13:31:33.181 DEBUG:teuthology.orchestra.run.smithi124:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2023-08-26T13:31:33.242 DEBUG:teuthology.orchestra.run.smithi124:> stat /dev/vg_nvme/lv_3 2023-08-26T13:31:33.302 INFO:teuthology.orchestra.run.smithi124.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2023-08-26T13:31:33.302 INFO:teuthology.orchestra.run.smithi124.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-08-26T13:31:33.303 INFO:teuthology.orchestra.run.smithi124.stdout:Device: 6h/6d Inode: 389527 Links: 1 2023-08-26T13:31:33.303 INFO:teuthology.orchestra.run.smithi124.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-08-26T13:31:33.303 INFO:teuthology.orchestra.run.smithi124.stdout:Context: system_u:object_r:device_t:s0 2023-08-26T13:31:33.303 INFO:teuthology.orchestra.run.smithi124.stdout:Access: 2023-08-26 13:30:30.231128922 +0000 2023-08-26T13:31:33.303 INFO:teuthology.orchestra.run.smithi124.stdout:Modify: 2023-08-26 13:24:30.042834763 +0000 2023-08-26T13:31:33.304 INFO:teuthology.orchestra.run.smithi124.stdout:Change: 2023-08-26 13:24:30.042834763 +0000 2023-08-26T13:31:33.304 INFO:teuthology.orchestra.run.smithi124.stdout: Birth: - 2023-08-26T13:31:33.304 DEBUG:teuthology.orchestra.run.smithi124:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2023-08-26T13:31:33.372 INFO:teuthology.orchestra.run.smithi124.stderr:1+0 records in 2023-08-26T13:31:33.372 INFO:teuthology.orchestra.run.smithi124.stderr:1+0 records out 2023-08-26T13:31:33.373 INFO:teuthology.orchestra.run.smithi124.stderr:512 bytes copied, 0.000241117 s, 2.1 MB/s 2023-08-26T13:31:33.374 DEBUG:teuthology.orchestra.run.smithi124:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2023-08-26T13:31:33.433 DEBUG:teuthology.orchestra.run.smithi124:> stat /dev/vg_nvme/lv_4 2023-08-26T13:31:33.491 INFO:teuthology.orchestra.run.smithi124.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2023-08-26T13:31:33.491 INFO:teuthology.orchestra.run.smithi124.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-08-26T13:31:33.491 INFO:teuthology.orchestra.run.smithi124.stdout:Device: 6h/6d Inode: 387068 Links: 1 2023-08-26T13:31:33.492 INFO:teuthology.orchestra.run.smithi124.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-08-26T13:31:33.492 INFO:teuthology.orchestra.run.smithi124.stdout:Context: system_u:object_r:device_t:s0 2023-08-26T13:31:33.492 INFO:teuthology.orchestra.run.smithi124.stdout:Access: 2023-08-26 13:30:30.231128922 +0000 2023-08-26T13:31:33.492 INFO:teuthology.orchestra.run.smithi124.stdout:Modify: 2023-08-26 13:24:30.544826150 +0000 2023-08-26T13:31:33.492 INFO:teuthology.orchestra.run.smithi124.stdout:Change: 2023-08-26 13:24:30.544826150 +0000 2023-08-26T13:31:33.493 INFO:teuthology.orchestra.run.smithi124.stdout: Birth: - 2023-08-26T13:31:33.493 DEBUG:teuthology.orchestra.run.smithi124:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2023-08-26T13:31:33.559 INFO:teuthology.orchestra.run.smithi124.stderr:1+0 records in 2023-08-26T13:31:33.559 INFO:teuthology.orchestra.run.smithi124.stderr:1+0 records out 2023-08-26T13:31:33.560 INFO:teuthology.orchestra.run.smithi124.stderr:512 bytes copied, 0.000204136 s, 2.5 MB/s 2023-08-26T13:31:33.561 DEBUG:teuthology.orchestra.run.smithi124:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2023-08-26T13:31:33.619 INFO:tasks.ceph:osd dev map: {'cluster2.osd.0': '/dev/vg_nvme/lv_1', 'cluster2.osd.1': '/dev/vg_nvme/lv_2', 'cluster2.osd.2': '/dev/vg_nvme/lv_3'} 2023-08-26T13:31:33.620 INFO:tasks.ceph:remote_to_roles_to_devs: {Remote(name='ubuntu@smithi124.front.sepia.ceph.com'): {'cluster2.osd.0': '/dev/vg_nvme/lv_1', 'cluster2.osd.1': '/dev/vg_nvme/lv_2', 'cluster2.osd.2': '/dev/vg_nvme/lv_3'}} 2023-08-26T13:31:33.620 INFO:tasks.ceph:Generating config... 2023-08-26T13:31:33.621 INFO:tasks.ceph:[client] debug rbd = 20 2023-08-26T13:31:33.621 INFO:tasks.ceph:[client] debug rbd_mirror = 15 2023-08-26T13:31:33.622 INFO:tasks.ceph:[client] log to stderr = False 2023-08-26T13:31:33.622 INFO:tasks.ceph:[client] rbd default features = 125 2023-08-26T13:31:33.622 INFO:tasks.ceph:[client] rbd mirror image policy type = simple 2023-08-26T13:31:33.623 INFO:tasks.ceph:[client.mirror.0] admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2023-08-26T13:31:33.623 INFO:tasks.ceph:[client.mirror.0] pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2023-08-26T13:31:33.623 INFO:tasks.ceph:[client.mirror.1] admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2023-08-26T13:31:33.623 INFO:tasks.ceph:[client.mirror.1] pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2023-08-26T13:31:33.624 INFO:tasks.ceph:[client.mirror.2] admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2023-08-26T13:31:33.624 INFO:tasks.ceph:[client.mirror.2] pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2023-08-26T13:31:33.624 INFO:tasks.ceph:[client.mirror.3] admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2023-08-26T13:31:33.624 INFO:tasks.ceph:[client.mirror.3] pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2023-08-26T13:31:33.625 INFO:tasks.ceph:[global] mon client directed command retry = 5 2023-08-26T13:31:33.625 INFO:tasks.ceph:[global] ms inject socket failures = 5000 2023-08-26T13:31:33.625 INFO:tasks.ceph:[mgr] debug mgr = 20 2023-08-26T13:31:33.625 INFO:tasks.ceph:[mgr] debug ms = 1 2023-08-26T13:31:33.626 INFO:tasks.ceph:[mon] debug mon = 20 2023-08-26T13:31:33.626 INFO:tasks.ceph:[mon] debug ms = 1 2023-08-26T13:31:33.626 INFO:tasks.ceph:[mon] debug paxos = 20 2023-08-26T13:31:33.626 INFO:tasks.ceph:[osd] bdev async discard = True 2023-08-26T13:31:33.627 INFO:tasks.ceph:[osd] bdev enable discard = True 2023-08-26T13:31:33.627 INFO:tasks.ceph:[osd] bluestore allocator = stupid 2023-08-26T13:31:33.627 INFO:tasks.ceph:[osd] bluestore block size = 96636764160 2023-08-26T13:31:33.627 INFO:tasks.ceph:[osd] bluestore fsck on mount = True 2023-08-26T13:31:33.628 INFO:tasks.ceph:[osd] debug bluefs = 1/20 2023-08-26T13:31:33.628 INFO:tasks.ceph:[osd] debug bluestore = 1/20 2023-08-26T13:31:33.628 INFO:tasks.ceph:[osd] debug ms = 1 2023-08-26T13:31:33.628 INFO:tasks.ceph:[osd] debug osd = 20 2023-08-26T13:31:33.629 INFO:tasks.ceph:[osd] debug rocksdb = 4/10 2023-08-26T13:31:33.629 INFO:tasks.ceph:[osd] mon osd backfillfull_ratio = 0.85 2023-08-26T13:31:33.629 INFO:tasks.ceph:[osd] mon osd full ratio = 0.9 2023-08-26T13:31:33.629 INFO:tasks.ceph:[osd] mon osd nearfull ratio = 0.8 2023-08-26T13:31:33.630 INFO:tasks.ceph:[osd] osd failsafe full ratio = 0.95 2023-08-26T13:31:33.630 INFO:tasks.ceph:[osd] osd objectstore = bluestore 2023-08-26T13:31:33.630 INFO:tasks.ceph:Setting up cluster2.mon.a... 2023-08-26T13:31:33.631 DEBUG:teuthology.orchestra.run.smithi124:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring /etc/ceph/cluster2.keyring 2023-08-26T13:31:33.703 INFO:teuthology.orchestra.run.smithi124.stdout:creating /etc/ceph/cluster2.keyring 2023-08-26T13:31:33.706 DEBUG:teuthology.orchestra.run.smithi124:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=mon. /etc/ceph/cluster2.keyring 2023-08-26T13:31:33.791 DEBUG:teuthology.orchestra.run.smithi124:> sudo chmod 0644 /etc/ceph/cluster2.keyring 2023-08-26T13:31:33.817 DEBUG:tasks.ceph:Ceph mon addresses: [('cluster2.mon.a', '172.21.15.124')] 2023-08-26T13:31:33.817 DEBUG:tasks.ceph:writing out conf {'global': {'chdir': '', 'pid file': '/var/run/ceph/$cluster-$name.pid', 'auth supported': 'cephx', 'filestore xattr use omap': 'true', 'mon clock drift allowed': '1.000', 'osd crush chooseleaf type': '0', 'auth debug': 'true', 'ms die on old message': 'true', 'ms die on bug': 'true', 'mon max pg per osd': '10000', 'mon pg warn max object skew': '0', 'osd_pool_default_pg_autoscale_mode': 'off', 'osd pool default size': '2', 'mon osd allow primary affinity': 'true', 'mon osd allow pg remap': 'true', 'mon warn on legacy crush tunables': 'false', 'mon warn on crush straw calc version zero': 'false', 'mon warn on no sortbitwise': 'false', 'mon warn on osd down out interval zero': 'false', 'mon warn on too few osds': 'false', 'mon_warn_on_pool_pg_num_not_power_of_two': 'false', 'mon_warn_on_pool_no_redundancy': 'false', 'mon_allow_pool_size_one': 'true', 'osd pool default erasure code profile': 'plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd', 'osd default data pool replay window': '5', 'mon allow pool delete': 'true', 'mon cluster log file level': 'debug', 'debug asserts on shutdown': 'true', 'mon health detail to clog': 'false', 'mon host': '172.21.15.124', 'mon client directed command retry': 5, 'ms inject socket failures': 5000}, 'osd': {'osd journal size': '100', 'osd scrub load threshold': '5.0', 'osd scrub max interval': '600', 'osd mclock profile': 'high_recovery_ops', 'osd recover clone overlap': 'true', 'osd recovery max chunk': '1048576', 'osd debug shutdown': 'true', 'osd debug op order': 'true', 'osd debug verify stray on activate': 'true', 'osd open classes on start': 'true', 'osd debug pg log writeout': 'true', 'osd deep scrub update digest min age': '30', 'osd map max advance': '10', 'journal zero on create': 'true', 'filestore ondisk finisher threads': '3', 'filestore apply finisher threads': '3', 'bdev debug aio': 'true', 'osd debug misdirected ops': 'true', 'bdev async discard': True, 'bdev enable discard': True, 'bluestore allocator': 'stupid', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore'}, 'mgr': {'debug ms': 1, 'debug mgr': 20, 'debug mon': '20', 'debug auth': '20', 'mon reweight min pgs per osd': '4', 'mon reweight min bytes per osd': '10', 'mgr/telemetry/nag': 'false'}, 'mon': {'debug ms': 1, 'debug mon': 20, 'debug paxos': 20, 'debug auth': '20', 'mon data avail warn': '5', 'mon mgr mkfs grace': '240', 'mon reweight min pgs per osd': '4', 'mon osd reporter subtree level': 'osd', 'mon osd prime pg temp': 'true', 'mon reweight min bytes per osd': '10', 'auth mon ticket ttl': '660', 'auth service ticket ttl': '240', 'mon_warn_on_insecure_global_id_reclaim': 'false', 'mon_warn_on_insecure_global_id_reclaim_allowed': 'false', 'mon_down_mkfs_grace': '2m', 'mon_warn_on_filestore_osds': 'false'}, 'client': {'rgw cache enabled': 'true', 'rgw enable ops log': 'true', 'rgw enable usage log': 'true', 'log file': '/var/log/ceph/$cluster-$name.$pid.log', 'admin socket': '/var/run/ceph/$cluster-$name.$pid.asok', 'debug rbd': 20, 'debug rbd_mirror': 15, 'log to stderr': False, 'rbd default features': 125, 'rbd mirror image policy type': 'simple'}, 'mon.a': {}, 'client.mirror.0': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.1': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.2': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.3': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}} 2023-08-26T13:31:33.818 DEBUG:teuthology.orchestra.run.smithi124:> set -ex 2023-08-26T13:31:33.819 DEBUG:teuthology.orchestra.run.smithi124:> dd of=/home/ubuntu/cephtest/ceph.tmp.conf 2023-08-26T13:31:33.873 DEBUG:teuthology.orchestra.run.smithi124:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage monmaptool -c /home/ubuntu/cephtest/ceph.tmp.conf --create --clobber --enable-all-features --add a 172.21.15.124 --print /home/ubuntu/cephtest/cluster2.monmap 2023-08-26T13:31:33.957 INFO:teuthology.orchestra.run.smithi124.stdout:monmaptool: monmap file /home/ubuntu/cephtest/cluster2.monmap 2023-08-26T13:31:33.957 INFO:teuthology.orchestra.run.smithi124.stdout:monmaptool: generated fsid 95df1446-13b8-484a-b4b7-3d1f11d3b887 2023-08-26T13:31:33.957 INFO:teuthology.orchestra.run.smithi124.stdout:setting min_mon_release = pacific 2023-08-26T13:31:33.958 INFO:teuthology.orchestra.run.smithi124.stdout:epoch 0 2023-08-26T13:31:33.958 INFO:teuthology.orchestra.run.smithi124.stdout:fsid 95df1446-13b8-484a-b4b7-3d1f11d3b887 2023-08-26T13:31:33.958 INFO:teuthology.orchestra.run.smithi124.stdout:last_changed 2023-08-26T13:31:33.956677+0000 2023-08-26T13:31:33.958 INFO:teuthology.orchestra.run.smithi124.stdout:created 2023-08-26T13:31:33.956677+0000 2023-08-26T13:31:33.958 INFO:teuthology.orchestra.run.smithi124.stdout:min_mon_release 16 (pacific) 2023-08-26T13:31:33.959 INFO:teuthology.orchestra.run.smithi124.stdout:election_strategy: 1 2023-08-26T13:31:33.959 INFO:teuthology.orchestra.run.smithi124.stdout:0: [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] mon.a 2023-08-26T13:31:33.959 INFO:teuthology.orchestra.run.smithi124.stdout:monmaptool: writing epoch 0 to /home/ubuntu/cephtest/cluster2.monmap (1 monitors) 2023-08-26T13:31:33.960 DEBUG:teuthology.orchestra.run.smithi124:> rm -- /home/ubuntu/cephtest/ceph.tmp.conf 2023-08-26T13:31:34.016 INFO:tasks.ceph:Writing /etc/ceph/cluster2.conf for FSID 95df1446-13b8-484a-b4b7-3d1f11d3b887... 2023-08-26T13:31:34.017 DEBUG:teuthology.orchestra.run.smithi031:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/cluster2.conf && sudo chmod 0644 /etc/ceph/cluster2.conf > /dev/null 2023-08-26T13:31:34.020 DEBUG:teuthology.orchestra.run.smithi124:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/cluster2.conf && sudo chmod 0644 /etc/ceph/cluster2.conf > /dev/null 2023-08-26T13:31:34.074 INFO:teuthology.orchestra.run.smithi031.stdout:[global] 2023-08-26T13:31:34.075 INFO:teuthology.orchestra.run.smithi031.stdout: chdir = "" 2023-08-26T13:31:34.075 INFO:teuthology.orchestra.run.smithi031.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2023-08-26T13:31:34.075 INFO:teuthology.orchestra.run.smithi031.stdout: auth supported = cephx 2023-08-26T13:31:34.076 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-08-26T13:31:34.076 INFO:teuthology.orchestra.run.smithi031.stdout: filestore xattr use omap = true 2023-08-26T13:31:34.076 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-08-26T13:31:34.076 INFO:teuthology.orchestra.run.smithi031.stdout: mon clock drift allowed = 1.000 2023-08-26T13:31:34.076 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-08-26T13:31:34.077 INFO:teuthology.orchestra.run.smithi031.stdout: osd crush chooseleaf type = 0 2023-08-26T13:31:34.077 INFO:teuthology.orchestra.run.smithi031.stdout: auth debug = true 2023-08-26T13:31:34.077 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-08-26T13:31:34.077 INFO:teuthology.orchestra.run.smithi031.stdout: ms die on old message = true 2023-08-26T13:31:34.077 INFO:teuthology.orchestra.run.smithi031.stdout: ms die on bug = true 2023-08-26T13:31:34.077 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-08-26T13:31:34.078 INFO:teuthology.orchestra.run.smithi031.stdout: mon max pg per osd = 10000 # >= luminous 2023-08-26T13:31:34.078 INFO:teuthology.orchestra.run.smithi031.stdout: mon pg warn max object skew = 0 2023-08-26T13:31:34.078 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-08-26T13:31:34.078 INFO:teuthology.orchestra.run.smithi031.stdout: # disable pg_autoscaler by default for new pools 2023-08-26T13:31:34.078 INFO:teuthology.orchestra.run.smithi031.stdout: osd_pool_default_pg_autoscale_mode = off 2023-08-26T13:31:34.079 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-08-26T13:31:34.079 INFO:teuthology.orchestra.run.smithi031.stdout: osd pool default size = 2 2023-08-26T13:31:34.079 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-08-26T13:31:34.079 INFO:teuthology.orchestra.run.smithi031.stdout: mon osd allow primary affinity = true 2023-08-26T13:31:34.079 INFO:teuthology.orchestra.run.smithi031.stdout: mon osd allow pg remap = true 2023-08-26T13:31:34.080 INFO:teuthology.orchestra.run.smithi031.stdout: mon warn on legacy crush tunables = false 2023-08-26T13:31:34.080 INFO:teuthology.orchestra.run.smithi031.stdout: mon warn on crush straw calc version zero = false 2023-08-26T13:31:34.080 INFO:teuthology.orchestra.run.smithi031.stdout: mon warn on no sortbitwise = false 2023-08-26T13:31:34.080 INFO:teuthology.orchestra.run.smithi031.stdout: mon warn on osd down out interval zero = false 2023-08-26T13:31:34.080 INFO:teuthology.orchestra.run.smithi031.stdout: mon warn on too few osds = false 2023-08-26T13:31:34.080 INFO:teuthology.orchestra.run.smithi031.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2023-08-26T13:31:34.081 INFO:teuthology.orchestra.run.smithi031.stdout: mon_warn_on_pool_no_redundancy = false 2023-08-26T13:31:34.081 INFO:teuthology.orchestra.run.smithi031.stdout: mon_allow_pool_size_one = true 2023-08-26T13:31:34.081 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-08-26T13:31:34.081 INFO:teuthology.orchestra.run.smithi031.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2023-08-26T13:31:34.081 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-08-26T13:31:34.082 INFO:teuthology.orchestra.run.smithi031.stdout: osd default data pool replay window = 5 2023-08-26T13:31:34.082 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-08-26T13:31:34.082 INFO:teuthology.orchestra.run.smithi031.stdout: mon allow pool delete = true 2023-08-26T13:31:34.082 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-08-26T13:31:34.082 INFO:teuthology.orchestra.run.smithi031.stdout: mon cluster log file level = debug 2023-08-26T13:31:34.082 INFO:teuthology.orchestra.run.smithi031.stdout: debug asserts on shutdown = true 2023-08-26T13:31:34.083 INFO:teuthology.orchestra.run.smithi031.stdout: mon health detail to clog = false 2023-08-26T13:31:34.083 INFO:teuthology.orchestra.run.smithi031.stdout: mon host = 172.21.15.124 2023-08-26T13:31:34.083 INFO:teuthology.orchestra.run.smithi031.stdout: mon client directed command retry = 5 2023-08-26T13:31:34.083 INFO:teuthology.orchestra.run.smithi031.stdout: ms inject socket failures = 5000 2023-08-26T13:31:34.083 INFO:teuthology.orchestra.run.smithi031.stdout: fsid = 95df1446-13b8-484a-b4b7-3d1f11d3b887 2023-08-26T13:31:34.084 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-08-26T13:31:34.084 INFO:teuthology.orchestra.run.smithi031.stdout:[osd] 2023-08-26T13:31:34.084 INFO:teuthology.orchestra.run.smithi031.stdout: osd journal size = 100 2023-08-26T13:31:34.084 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-08-26T13:31:34.084 INFO:teuthology.orchestra.run.smithi031.stdout: osd scrub load threshold = 5.0 2023-08-26T13:31:34.085 INFO:teuthology.orchestra.run.smithi031.stdout: osd scrub max interval = 600 2023-08-26T13:31:34.085 INFO:teuthology.orchestra.run.smithi031.stdout: osd mclock profile = high_recovery_ops 2023-08-26T13:31:34.085 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-08-26T13:31:34.085 INFO:teuthology.orchestra.run.smithi031.stdout: osd recover clone overlap = true 2023-08-26T13:31:34.085 INFO:teuthology.orchestra.run.smithi031.stdout: osd recovery max chunk = 1048576 2023-08-26T13:31:34.086 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-08-26T13:31:34.086 INFO:teuthology.orchestra.run.smithi031.stdout: osd debug shutdown = true 2023-08-26T13:31:34.086 INFO:teuthology.orchestra.run.smithi031.stdout: osd debug op order = true 2023-08-26T13:31:34.086 INFO:teuthology.orchestra.run.smithi031.stdout: osd debug verify stray on activate = true 2023-08-26T13:31:34.086 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-08-26T13:31:34.086 INFO:teuthology.orchestra.run.smithi031.stdout: osd open classes on start = true 2023-08-26T13:31:34.087 INFO:teuthology.orchestra.run.smithi031.stdout: osd debug pg log writeout = true 2023-08-26T13:31:34.087 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-08-26T13:31:34.087 INFO:teuthology.orchestra.run.smithi031.stdout: osd deep scrub update digest min age = 30 2023-08-26T13:31:34.087 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-08-26T13:31:34.087 INFO:teuthology.orchestra.run.smithi031.stdout: osd map max advance = 10 2023-08-26T13:31:34.088 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-08-26T13:31:34.088 INFO:teuthology.orchestra.run.smithi031.stdout: journal zero on create = true 2023-08-26T13:31:34.088 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-08-26T13:31:34.088 INFO:teuthology.orchestra.run.smithi031.stdout: filestore ondisk finisher threads = 3 2023-08-26T13:31:34.088 INFO:teuthology.orchestra.run.smithi031.stdout: filestore apply finisher threads = 3 2023-08-26T13:31:34.088 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-08-26T13:31:34.089 INFO:teuthology.orchestra.run.smithi031.stdout: bdev debug aio = true 2023-08-26T13:31:34.089 INFO:teuthology.orchestra.run.smithi031.stdout: osd debug misdirected ops = true 2023-08-26T13:31:34.089 INFO:teuthology.orchestra.run.smithi031.stdout: bdev async discard = True 2023-08-26T13:31:34.089 INFO:teuthology.orchestra.run.smithi031.stdout: bdev enable discard = True 2023-08-26T13:31:34.089 INFO:teuthology.orchestra.run.smithi031.stdout: bluestore allocator = stupid 2023-08-26T13:31:34.090 INFO:teuthology.orchestra.run.smithi031.stdout: bluestore block size = 96636764160 2023-08-26T13:31:34.090 INFO:teuthology.orchestra.run.smithi031.stdout: bluestore fsck on mount = True 2023-08-26T13:31:34.090 INFO:teuthology.orchestra.run.smithi031.stdout: debug bluefs = 1/20 2023-08-26T13:31:34.090 INFO:teuthology.orchestra.run.smithi031.stdout: debug bluestore = 1/20 2023-08-26T13:31:34.090 INFO:teuthology.orchestra.run.smithi031.stdout: debug ms = 1 2023-08-26T13:31:34.090 INFO:teuthology.orchestra.run.smithi031.stdout: debug osd = 20 2023-08-26T13:31:34.091 INFO:teuthology.orchestra.run.smithi031.stdout: debug rocksdb = 4/10 2023-08-26T13:31:34.091 INFO:teuthology.orchestra.run.smithi031.stdout: mon osd backfillfull_ratio = 0.85 2023-08-26T13:31:34.091 INFO:teuthology.orchestra.run.smithi031.stdout: mon osd full ratio = 0.9 2023-08-26T13:31:34.091 INFO:teuthology.orchestra.run.smithi031.stdout: mon osd nearfull ratio = 0.8 2023-08-26T13:31:34.091 INFO:teuthology.orchestra.run.smithi031.stdout: osd failsafe full ratio = 0.95 2023-08-26T13:31:34.092 INFO:teuthology.orchestra.run.smithi031.stdout: osd objectstore = bluestore 2023-08-26T13:31:34.092 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-08-26T13:31:34.092 INFO:teuthology.orchestra.run.smithi031.stdout:[mgr] 2023-08-26T13:31:34.092 INFO:teuthology.orchestra.run.smithi031.stdout: debug ms = 1 2023-08-26T13:31:34.092 INFO:teuthology.orchestra.run.smithi031.stdout: debug mgr = 20 2023-08-26T13:31:34.092 INFO:teuthology.orchestra.run.smithi031.stdout: debug mon = 20 2023-08-26T13:31:34.093 INFO:teuthology.orchestra.run.smithi031.stdout: debug auth = 20 2023-08-26T13:31:34.093 INFO:teuthology.orchestra.run.smithi031.stdout: mon reweight min pgs per osd = 4 2023-08-26T13:31:34.093 INFO:teuthology.orchestra.run.smithi031.stdout: mon reweight min bytes per osd = 10 2023-08-26T13:31:34.093 INFO:teuthology.orchestra.run.smithi031.stdout: mgr/telemetry/nag = false 2023-08-26T13:31:34.093 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-08-26T13:31:34.093 INFO:teuthology.orchestra.run.smithi031.stdout:[mon] 2023-08-26T13:31:34.094 INFO:teuthology.orchestra.run.smithi031.stdout: debug ms = 1 2023-08-26T13:31:34.094 INFO:teuthology.orchestra.run.smithi031.stdout: debug mon = 20 2023-08-26T13:31:34.094 INFO:teuthology.orchestra.run.smithi031.stdout: debug paxos = 20 2023-08-26T13:31:34.094 INFO:teuthology.orchestra.run.smithi031.stdout: debug auth = 20 2023-08-26T13:31:34.094 INFO:teuthology.orchestra.run.smithi031.stdout: mon data avail warn = 5 2023-08-26T13:31:34.094 INFO:teuthology.orchestra.run.smithi031.stdout: mon mgr mkfs grace = 240 2023-08-26T13:31:34.095 INFO:teuthology.orchestra.run.smithi031.stdout: mon reweight min pgs per osd = 4 2023-08-26T13:31:34.095 INFO:teuthology.orchestra.run.smithi031.stdout: mon osd reporter subtree level = osd 2023-08-26T13:31:34.095 INFO:teuthology.orchestra.run.smithi031.stdout: mon osd prime pg temp = true 2023-08-26T13:31:34.095 INFO:teuthology.orchestra.run.smithi031.stdout: mon reweight min bytes per osd = 10 2023-08-26T13:31:34.095 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-08-26T13:31:34.096 INFO:teuthology.orchestra.run.smithi031.stdout: # rotate auth tickets quickly to exercise renewal paths 2023-08-26T13:31:34.096 INFO:teuthology.orchestra.run.smithi031.stdout: auth mon ticket ttl = 660 # 11m 2023-08-26T13:31:34.096 INFO:teuthology.orchestra.run.smithi031.stdout: auth service ticket ttl = 240 # 4m 2023-08-26T13:31:34.096 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-08-26T13:31:34.096 INFO:teuthology.orchestra.run.smithi031.stdout: # don't complain about insecure global_id in the test suite 2023-08-26T13:31:34.097 INFO:teuthology.orchestra.run.smithi031.stdout: mon_warn_on_insecure_global_id_reclaim = false 2023-08-26T13:31:34.097 INFO:teuthology.orchestra.run.smithi031.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2023-08-26T13:31:34.097 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-08-26T13:31:34.097 INFO:teuthology.orchestra.run.smithi031.stdout: # 1m isn't quite enough 2023-08-26T13:31:34.097 INFO:teuthology.orchestra.run.smithi031.stdout: mon_down_mkfs_grace = 2m 2023-08-26T13:31:34.097 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-08-26T13:31:34.098 INFO:teuthology.orchestra.run.smithi031.stdout: mon_warn_on_filestore_osds = false 2023-08-26T13:31:34.098 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-08-26T13:31:34.098 INFO:teuthology.orchestra.run.smithi031.stdout:[client] 2023-08-26T13:31:34.098 INFO:teuthology.orchestra.run.smithi031.stdout: rgw cache enabled = true 2023-08-26T13:31:34.098 INFO:teuthology.orchestra.run.smithi031.stdout: rgw enable ops log = true 2023-08-26T13:31:34.098 INFO:teuthology.orchestra.run.smithi031.stdout: rgw enable usage log = true 2023-08-26T13:31:34.099 INFO:teuthology.orchestra.run.smithi031.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2023-08-26T13:31:34.099 INFO:teuthology.orchestra.run.smithi031.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2023-08-26T13:31:34.099 INFO:teuthology.orchestra.run.smithi031.stdout: debug rbd = 20 2023-08-26T13:31:34.099 INFO:teuthology.orchestra.run.smithi031.stdout: debug rbd_mirror = 15 2023-08-26T13:31:34.099 INFO:teuthology.orchestra.run.smithi031.stdout: log to stderr = False 2023-08-26T13:31:34.099 INFO:teuthology.orchestra.run.smithi031.stdout: rbd default features = 125 2023-08-26T13:31:34.100 INFO:teuthology.orchestra.run.smithi031.stdout: rbd mirror image policy type = simple 2023-08-26T13:31:34.100 INFO:teuthology.orchestra.run.smithi031.stdout:[mon.a] 2023-08-26T13:31:34.100 INFO:teuthology.orchestra.run.smithi031.stdout:[client.mirror.0] 2023-08-26T13:31:34.100 INFO:teuthology.orchestra.run.smithi031.stdout: admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2023-08-26T13:31:34.100 INFO:teuthology.orchestra.run.smithi031.stdout: pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2023-08-26T13:31:34.101 INFO:teuthology.orchestra.run.smithi031.stdout:[client.mirror.1] 2023-08-26T13:31:34.101 INFO:teuthology.orchestra.run.smithi031.stdout: admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2023-08-26T13:31:34.101 INFO:teuthology.orchestra.run.smithi031.stdout: pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2023-08-26T13:31:34.101 INFO:teuthology.orchestra.run.smithi031.stdout:[client.mirror.2] 2023-08-26T13:31:34.101 INFO:teuthology.orchestra.run.smithi031.stdout: admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2023-08-26T13:31:34.101 INFO:teuthology.orchestra.run.smithi031.stdout: pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2023-08-26T13:31:34.102 INFO:teuthology.orchestra.run.smithi031.stdout:[client.mirror.3] 2023-08-26T13:31:34.102 INFO:teuthology.orchestra.run.smithi031.stdout: admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2023-08-26T13:31:34.102 INFO:teuthology.orchestra.run.smithi031.stdout: pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2023-08-26T13:31:34.103 INFO:teuthology.orchestra.run.smithi124.stdout:[global] 2023-08-26T13:31:34.103 INFO:teuthology.orchestra.run.smithi124.stdout: chdir = "" 2023-08-26T13:31:34.103 INFO:teuthology.orchestra.run.smithi124.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2023-08-26T13:31:34.103 INFO:teuthology.orchestra.run.smithi124.stdout: auth supported = cephx 2023-08-26T13:31:34.103 INFO:teuthology.orchestra.run.smithi124.stdout: 2023-08-26T13:31:34.104 INFO:teuthology.orchestra.run.smithi124.stdout: filestore xattr use omap = true 2023-08-26T13:31:34.104 INFO:teuthology.orchestra.run.smithi124.stdout: 2023-08-26T13:31:34.104 INFO:teuthology.orchestra.run.smithi124.stdout: mon clock drift allowed = 1.000 2023-08-26T13:31:34.104 INFO:teuthology.orchestra.run.smithi124.stdout: 2023-08-26T13:31:34.104 INFO:teuthology.orchestra.run.smithi124.stdout: osd crush chooseleaf type = 0 2023-08-26T13:31:34.105 INFO:teuthology.orchestra.run.smithi124.stdout: auth debug = true 2023-08-26T13:31:34.105 INFO:teuthology.orchestra.run.smithi124.stdout: 2023-08-26T13:31:34.105 INFO:teuthology.orchestra.run.smithi124.stdout: ms die on old message = true 2023-08-26T13:31:34.105 INFO:teuthology.orchestra.run.smithi124.stdout: ms die on bug = true 2023-08-26T13:31:34.105 INFO:teuthology.orchestra.run.smithi124.stdout: 2023-08-26T13:31:34.105 INFO:teuthology.orchestra.run.smithi124.stdout: mon max pg per osd = 10000 # >= luminous 2023-08-26T13:31:34.106 INFO:teuthology.orchestra.run.smithi124.stdout: mon pg warn max object skew = 0 2023-08-26T13:31:34.106 INFO:teuthology.orchestra.run.smithi124.stdout: 2023-08-26T13:31:34.106 INFO:teuthology.orchestra.run.smithi124.stdout: # disable pg_autoscaler by default for new pools 2023-08-26T13:31:34.106 INFO:teuthology.orchestra.run.smithi124.stdout: osd_pool_default_pg_autoscale_mode = off 2023-08-26T13:31:34.106 INFO:teuthology.orchestra.run.smithi124.stdout: 2023-08-26T13:31:34.106 INFO:teuthology.orchestra.run.smithi124.stdout: osd pool default size = 2 2023-08-26T13:31:34.107 INFO:teuthology.orchestra.run.smithi124.stdout: 2023-08-26T13:31:34.107 INFO:teuthology.orchestra.run.smithi124.stdout: mon osd allow primary affinity = true 2023-08-26T13:31:34.107 INFO:teuthology.orchestra.run.smithi124.stdout: mon osd allow pg remap = true 2023-08-26T13:31:34.107 INFO:teuthology.orchestra.run.smithi124.stdout: mon warn on legacy crush tunables = false 2023-08-26T13:31:34.107 INFO:teuthology.orchestra.run.smithi124.stdout: mon warn on crush straw calc version zero = false 2023-08-26T13:31:34.108 INFO:teuthology.orchestra.run.smithi124.stdout: mon warn on no sortbitwise = false 2023-08-26T13:31:34.108 INFO:teuthology.orchestra.run.smithi124.stdout: mon warn on osd down out interval zero = false 2023-08-26T13:31:34.108 INFO:teuthology.orchestra.run.smithi124.stdout: mon warn on too few osds = false 2023-08-26T13:31:34.108 INFO:teuthology.orchestra.run.smithi124.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2023-08-26T13:31:34.108 INFO:teuthology.orchestra.run.smithi124.stdout: mon_warn_on_pool_no_redundancy = false 2023-08-26T13:31:34.108 INFO:teuthology.orchestra.run.smithi124.stdout: mon_allow_pool_size_one = true 2023-08-26T13:31:34.109 INFO:teuthology.orchestra.run.smithi124.stdout: 2023-08-26T13:31:34.109 INFO:teuthology.orchestra.run.smithi124.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2023-08-26T13:31:34.109 INFO:teuthology.orchestra.run.smithi124.stdout: 2023-08-26T13:31:34.109 INFO:teuthology.orchestra.run.smithi124.stdout: osd default data pool replay window = 5 2023-08-26T13:31:34.109 INFO:teuthology.orchestra.run.smithi124.stdout: 2023-08-26T13:31:34.109 INFO:teuthology.orchestra.run.smithi124.stdout: mon allow pool delete = true 2023-08-26T13:31:34.110 INFO:teuthology.orchestra.run.smithi124.stdout: 2023-08-26T13:31:34.110 INFO:teuthology.orchestra.run.smithi124.stdout: mon cluster log file level = debug 2023-08-26T13:31:34.110 INFO:teuthology.orchestra.run.smithi124.stdout: debug asserts on shutdown = true 2023-08-26T13:31:34.110 INFO:teuthology.orchestra.run.smithi124.stdout: mon health detail to clog = false 2023-08-26T13:31:34.110 INFO:teuthology.orchestra.run.smithi124.stdout: mon host = 172.21.15.124 2023-08-26T13:31:34.110 INFO:teuthology.orchestra.run.smithi124.stdout: mon client directed command retry = 5 2023-08-26T13:31:34.111 INFO:teuthology.orchestra.run.smithi124.stdout: ms inject socket failures = 5000 2023-08-26T13:31:34.111 INFO:teuthology.orchestra.run.smithi124.stdout: fsid = 95df1446-13b8-484a-b4b7-3d1f11d3b887 2023-08-26T13:31:34.111 INFO:teuthology.orchestra.run.smithi124.stdout: 2023-08-26T13:31:34.111 INFO:teuthology.orchestra.run.smithi124.stdout:[osd] 2023-08-26T13:31:34.111 INFO:teuthology.orchestra.run.smithi124.stdout: osd journal size = 100 2023-08-26T13:31:34.112 INFO:teuthology.orchestra.run.smithi124.stdout: 2023-08-26T13:31:34.112 INFO:teuthology.orchestra.run.smithi124.stdout: osd scrub load threshold = 5.0 2023-08-26T13:31:34.112 INFO:teuthology.orchestra.run.smithi124.stdout: osd scrub max interval = 600 2023-08-26T13:31:34.112 INFO:teuthology.orchestra.run.smithi124.stdout: osd mclock profile = high_recovery_ops 2023-08-26T13:31:34.112 INFO:teuthology.orchestra.run.smithi124.stdout: 2023-08-26T13:31:34.112 INFO:teuthology.orchestra.run.smithi124.stdout: osd recover clone overlap = true 2023-08-26T13:31:34.113 INFO:teuthology.orchestra.run.smithi124.stdout: osd recovery max chunk = 1048576 2023-08-26T13:31:34.113 INFO:teuthology.orchestra.run.smithi124.stdout: 2023-08-26T13:31:34.113 INFO:teuthology.orchestra.run.smithi124.stdout: osd debug shutdown = true 2023-08-26T13:31:34.113 INFO:teuthology.orchestra.run.smithi124.stdout: osd debug op order = true 2023-08-26T13:31:34.113 INFO:teuthology.orchestra.run.smithi124.stdout: osd debug verify stray on activate = true 2023-08-26T13:31:34.113 INFO:teuthology.orchestra.run.smithi124.stdout: 2023-08-26T13:31:34.114 INFO:teuthology.orchestra.run.smithi124.stdout: osd open classes on start = true 2023-08-26T13:31:34.114 INFO:teuthology.orchestra.run.smithi124.stdout: osd debug pg log writeout = true 2023-08-26T13:31:34.114 INFO:teuthology.orchestra.run.smithi124.stdout: 2023-08-26T13:31:34.114 INFO:teuthology.orchestra.run.smithi124.stdout: osd deep scrub update digest min age = 30 2023-08-26T13:31:34.114 INFO:teuthology.orchestra.run.smithi124.stdout: 2023-08-26T13:31:34.115 INFO:teuthology.orchestra.run.smithi124.stdout: osd map max advance = 10 2023-08-26T13:31:34.115 INFO:teuthology.orchestra.run.smithi124.stdout: 2023-08-26T13:31:34.115 INFO:teuthology.orchestra.run.smithi124.stdout: journal zero on create = true 2023-08-26T13:31:34.115 INFO:teuthology.orchestra.run.smithi124.stdout: 2023-08-26T13:31:34.115 INFO:teuthology.orchestra.run.smithi124.stdout: filestore ondisk finisher threads = 3 2023-08-26T13:31:34.115 INFO:teuthology.orchestra.run.smithi124.stdout: filestore apply finisher threads = 3 2023-08-26T13:31:34.116 INFO:teuthology.orchestra.run.smithi124.stdout: 2023-08-26T13:31:34.116 INFO:teuthology.orchestra.run.smithi124.stdout: bdev debug aio = true 2023-08-26T13:31:34.116 INFO:teuthology.orchestra.run.smithi124.stdout: osd debug misdirected ops = true 2023-08-26T13:31:34.116 INFO:teuthology.orchestra.run.smithi124.stdout: bdev async discard = True 2023-08-26T13:31:34.116 INFO:teuthology.orchestra.run.smithi124.stdout: bdev enable discard = True 2023-08-26T13:31:34.117 INFO:teuthology.orchestra.run.smithi124.stdout: bluestore allocator = stupid 2023-08-26T13:31:34.117 INFO:teuthology.orchestra.run.smithi124.stdout: bluestore block size = 96636764160 2023-08-26T13:31:34.117 INFO:teuthology.orchestra.run.smithi124.stdout: bluestore fsck on mount = True 2023-08-26T13:31:34.117 INFO:teuthology.orchestra.run.smithi124.stdout: debug bluefs = 1/20 2023-08-26T13:31:34.117 INFO:teuthology.orchestra.run.smithi124.stdout: debug bluestore = 1/20 2023-08-26T13:31:34.118 INFO:teuthology.orchestra.run.smithi124.stdout: debug ms = 1 2023-08-26T13:31:34.118 INFO:teuthology.orchestra.run.smithi124.stdout: debug osd = 20 2023-08-26T13:31:34.118 INFO:teuthology.orchestra.run.smithi124.stdout: debug rocksdb = 4/10 2023-08-26T13:31:34.118 INFO:teuthology.orchestra.run.smithi124.stdout: mon osd backfillfull_ratio = 0.85 2023-08-26T13:31:34.118 INFO:teuthology.orchestra.run.smithi124.stdout: mon osd full ratio = 0.9 2023-08-26T13:31:34.119 INFO:teuthology.orchestra.run.smithi124.stdout: mon osd nearfull ratio = 0.8 2023-08-26T13:31:34.119 INFO:teuthology.orchestra.run.smithi124.stdout: osd failsafe full ratio = 0.95 2023-08-26T13:31:34.119 INFO:teuthology.orchestra.run.smithi124.stdout: osd objectstore = bluestore 2023-08-26T13:31:34.119 INFO:teuthology.orchestra.run.smithi124.stdout: 2023-08-26T13:31:34.119 INFO:teuthology.orchestra.run.smithi124.stdout:[mgr] 2023-08-26T13:31:34.120 INFO:teuthology.orchestra.run.smithi124.stdout: debug ms = 1 2023-08-26T13:31:34.120 INFO:teuthology.orchestra.run.smithi124.stdout: debug mgr = 20 2023-08-26T13:31:34.120 INFO:teuthology.orchestra.run.smithi124.stdout: debug mon = 20 2023-08-26T13:31:34.120 INFO:teuthology.orchestra.run.smithi124.stdout: debug auth = 20 2023-08-26T13:31:34.120 INFO:teuthology.orchestra.run.smithi124.stdout: mon reweight min pgs per osd = 4 2023-08-26T13:31:34.120 INFO:teuthology.orchestra.run.smithi124.stdout: mon reweight min bytes per osd = 10 2023-08-26T13:31:34.121 INFO:teuthology.orchestra.run.smithi124.stdout: mgr/telemetry/nag = false 2023-08-26T13:31:34.121 INFO:teuthology.orchestra.run.smithi124.stdout: 2023-08-26T13:31:34.121 INFO:teuthology.orchestra.run.smithi124.stdout:[mon] 2023-08-26T13:31:34.121 INFO:teuthology.orchestra.run.smithi124.stdout: debug ms = 1 2023-08-26T13:31:34.121 INFO:teuthology.orchestra.run.smithi124.stdout: debug mon = 20 2023-08-26T13:31:34.121 INFO:teuthology.orchestra.run.smithi124.stdout: debug paxos = 20 2023-08-26T13:31:34.122 INFO:teuthology.orchestra.run.smithi124.stdout: debug auth = 20 2023-08-26T13:31:34.122 INFO:teuthology.orchestra.run.smithi124.stdout: mon data avail warn = 5 2023-08-26T13:31:34.122 INFO:teuthology.orchestra.run.smithi124.stdout: mon mgr mkfs grace = 240 2023-08-26T13:31:34.122 INFO:teuthology.orchestra.run.smithi124.stdout: mon reweight min pgs per osd = 4 2023-08-26T13:31:34.122 INFO:teuthology.orchestra.run.smithi124.stdout: mon osd reporter subtree level = osd 2023-08-26T13:31:34.122 INFO:teuthology.orchestra.run.smithi124.stdout: mon osd prime pg temp = true 2023-08-26T13:31:34.123 INFO:teuthology.orchestra.run.smithi124.stdout: mon reweight min bytes per osd = 10 2023-08-26T13:31:34.123 INFO:teuthology.orchestra.run.smithi124.stdout: 2023-08-26T13:31:34.123 INFO:teuthology.orchestra.run.smithi124.stdout: # rotate auth tickets quickly to exercise renewal paths 2023-08-26T13:31:34.123 INFO:teuthology.orchestra.run.smithi124.stdout: auth mon ticket ttl = 660 # 11m 2023-08-26T13:31:34.123 INFO:teuthology.orchestra.run.smithi124.stdout: auth service ticket ttl = 240 # 4m 2023-08-26T13:31:34.124 INFO:teuthology.orchestra.run.smithi124.stdout: 2023-08-26T13:31:34.124 INFO:teuthology.orchestra.run.smithi124.stdout: # don't complain about insecure global_id in the test suite 2023-08-26T13:31:34.124 INFO:teuthology.orchestra.run.smithi124.stdout: mon_warn_on_insecure_global_id_reclaim = false 2023-08-26T13:31:34.124 INFO:teuthology.orchestra.run.smithi124.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2023-08-26T13:31:34.124 INFO:teuthology.orchestra.run.smithi124.stdout: 2023-08-26T13:31:34.125 INFO:teuthology.orchestra.run.smithi124.stdout: # 1m isn't quite enough 2023-08-26T13:31:34.125 INFO:teuthology.orchestra.run.smithi124.stdout: mon_down_mkfs_grace = 2m 2023-08-26T13:31:34.125 INFO:teuthology.orchestra.run.smithi124.stdout: 2023-08-26T13:31:34.125 INFO:teuthology.orchestra.run.smithi124.stdout: mon_warn_on_filestore_osds = false 2023-08-26T13:31:34.125 INFO:teuthology.orchestra.run.smithi124.stdout: 2023-08-26T13:31:34.125 INFO:teuthology.orchestra.run.smithi124.stdout:[client] 2023-08-26T13:31:34.126 INFO:teuthology.orchestra.run.smithi124.stdout: rgw cache enabled = true 2023-08-26T13:31:34.126 INFO:teuthology.orchestra.run.smithi124.stdout: rgw enable ops log = true 2023-08-26T13:31:34.126 INFO:teuthology.orchestra.run.smithi124.stdout: rgw enable usage log = true 2023-08-26T13:31:34.126 INFO:teuthology.orchestra.run.smithi124.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2023-08-26T13:31:34.126 INFO:teuthology.orchestra.run.smithi124.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2023-08-26T13:31:34.127 INFO:teuthology.orchestra.run.smithi124.stdout: debug rbd = 20 2023-08-26T13:31:34.127 INFO:teuthology.orchestra.run.smithi124.stdout: debug rbd_mirror = 15 2023-08-26T13:31:34.127 INFO:teuthology.orchestra.run.smithi124.stdout: log to stderr = False 2023-08-26T13:31:34.127 INFO:teuthology.orchestra.run.smithi124.stdout: rbd default features = 125 2023-08-26T13:31:34.127 INFO:teuthology.orchestra.run.smithi124.stdout: rbd mirror image policy type = simple 2023-08-26T13:31:34.127 INFO:teuthology.orchestra.run.smithi124.stdout:[mon.a] 2023-08-26T13:31:34.128 INFO:teuthology.orchestra.run.smithi124.stdout:[client.mirror.0] 2023-08-26T13:31:34.128 INFO:teuthology.orchestra.run.smithi124.stdout: admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2023-08-26T13:31:34.128 INFO:teuthology.orchestra.run.smithi124.stdout: pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2023-08-26T13:31:34.128 INFO:teuthology.orchestra.run.smithi124.stdout:[client.mirror.1] 2023-08-26T13:31:34.128 INFO:teuthology.orchestra.run.smithi124.stdout: admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2023-08-26T13:31:34.129 INFO:teuthology.orchestra.run.smithi124.stdout: pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2023-08-26T13:31:34.129 INFO:teuthology.orchestra.run.smithi124.stdout:[client.mirror.2] 2023-08-26T13:31:34.129 INFO:teuthology.orchestra.run.smithi124.stdout: admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2023-08-26T13:31:34.129 INFO:teuthology.orchestra.run.smithi124.stdout: pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2023-08-26T13:31:34.129 INFO:teuthology.orchestra.run.smithi124.stdout:[client.mirror.3] 2023-08-26T13:31:34.129 INFO:teuthology.orchestra.run.smithi124.stdout: admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2023-08-26T13:31:34.130 INFO:teuthology.orchestra.run.smithi124.stdout: pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2023-08-26T13:31:34.131 INFO:tasks.ceph:Creating admin key on cluster2.mon.a... 2023-08-26T13:31:34.131 DEBUG:teuthology.orchestra.run.smithi124:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=client.admin --cap mon 'allow *' --cap osd 'allow *' --cap mds 'allow *' --cap mgr 'allow *' /etc/ceph/cluster2.keyring 2023-08-26T13:31:34.188 INFO:tasks.ceph:Copying monmap to all nodes... 2023-08-26T13:31:34.189 DEBUG:teuthology.orchestra.run.smithi124:> set -ex 2023-08-26T13:31:34.189 DEBUG:teuthology.orchestra.run.smithi124:> dd if=/etc/ceph/cluster2.keyring of=/dev/stdout 2023-08-26T13:31:34.205 DEBUG:teuthology.orchestra.run.smithi124:> set -ex 2023-08-26T13:31:34.206 DEBUG:teuthology.orchestra.run.smithi124:> dd if=/home/ubuntu/cephtest/cluster2.monmap of=/dev/stdout 2023-08-26T13:31:34.262 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi031.front.sepia.ceph.com 2023-08-26T13:31:34.263 DEBUG:teuthology.orchestra.run.smithi031:> set -ex 2023-08-26T13:31:34.263 DEBUG:teuthology.orchestra.run.smithi031:> sudo dd of=/etc/ceph/cluster2.keyring 2023-08-26T13:31:34.263 DEBUG:teuthology.orchestra.run.smithi031:> sudo chmod 0644 /etc/ceph/cluster2.keyring 2023-08-26T13:31:34.309 DEBUG:teuthology.orchestra.run.smithi031:> set -ex 2023-08-26T13:31:34.310 DEBUG:teuthology.orchestra.run.smithi031:> dd of=/home/ubuntu/cephtest/cluster2.monmap 2023-08-26T13:31:34.369 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi124.front.sepia.ceph.com 2023-08-26T13:31:34.370 DEBUG:teuthology.orchestra.run.smithi124:> set -ex 2023-08-26T13:31:34.370 DEBUG:teuthology.orchestra.run.smithi124:> sudo dd of=/etc/ceph/cluster2.keyring 2023-08-26T13:31:34.370 DEBUG:teuthology.orchestra.run.smithi124:> sudo chmod 0644 /etc/ceph/cluster2.keyring 2023-08-26T13:31:34.407 DEBUG:teuthology.orchestra.run.smithi124:> set -ex 2023-08-26T13:31:34.407 DEBUG:teuthology.orchestra.run.smithi124:> dd of=/home/ubuntu/cephtest/cluster2.monmap 2023-08-26T13:31:34.463 INFO:tasks.ceph:Setting up mon nodes... 2023-08-26T13:31:34.463 INFO:tasks.ceph:Setting up mgr nodes... 2023-08-26T13:31:34.464 DEBUG:teuthology.orchestra.run.smithi031:> sudo mkdir -p /var/lib/ceph/mgr/cluster2-x && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.x /var/lib/ceph/mgr/cluster2-x/keyring 2023-08-26T13:31:34.525 INFO:teuthology.orchestra.run.smithi031.stdout:creating /var/lib/ceph/mgr/cluster2-x/keyring 2023-08-26T13:31:34.527 INFO:tasks.ceph:Setting up mds nodes... 2023-08-26T13:31:34.528 INFO:tasks.ceph_client:Setting up client nodes... 2023-08-26T13:31:34.528 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.0 /etc/ceph/cluster2.client.0.keyring && sudo chmod 0644 /etc/ceph/cluster2.client.0.keyring 2023-08-26T13:31:34.575 INFO:teuthology.orchestra.run.smithi031.stdout:creating /etc/ceph/cluster2.client.0.keyring 2023-08-26T13:31:34.588 DEBUG:teuthology.orchestra.run.smithi124:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.mirror /etc/ceph/cluster2.client.mirror.keyring && sudo chmod 0644 /etc/ceph/cluster2.client.mirror.keyring 2023-08-26T13:31:34.631 INFO:teuthology.orchestra.run.smithi124.stdout:creating /etc/ceph/cluster2.client.mirror.keyring 2023-08-26T13:31:34.644 DEBUG:teuthology.orchestra.run.smithi124:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.mirror.0 /etc/ceph/cluster2.client.mirror.0.keyring && sudo chmod 0644 /etc/ceph/cluster2.client.mirror.0.keyring 2023-08-26T13:31:34.727 INFO:teuthology.orchestra.run.smithi124.stdout:creating /etc/ceph/cluster2.client.mirror.0.keyring 2023-08-26T13:31:34.740 DEBUG:teuthology.orchestra.run.smithi124:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.mirror.1 /etc/ceph/cluster2.client.mirror.1.keyring && sudo chmod 0644 /etc/ceph/cluster2.client.mirror.1.keyring 2023-08-26T13:31:34.823 INFO:teuthology.orchestra.run.smithi124.stdout:creating /etc/ceph/cluster2.client.mirror.1.keyring 2023-08-26T13:31:34.836 DEBUG:teuthology.orchestra.run.smithi124:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.mirror.2 /etc/ceph/cluster2.client.mirror.2.keyring && sudo chmod 0644 /etc/ceph/cluster2.client.mirror.2.keyring 2023-08-26T13:31:34.918 INFO:teuthology.orchestra.run.smithi124.stdout:creating /etc/ceph/cluster2.client.mirror.2.keyring 2023-08-26T13:31:34.931 DEBUG:teuthology.orchestra.run.smithi124:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.mirror.3 /etc/ceph/cluster2.client.mirror.3.keyring && sudo chmod 0644 /etc/ceph/cluster2.client.mirror.3.keyring 2023-08-26T13:31:34.974 INFO:teuthology.orchestra.run.smithi124.stdout:creating /etc/ceph/cluster2.client.mirror.3.keyring 2023-08-26T13:31:34.987 DEBUG:teuthology.orchestra.run.smithi124:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.mirror.4 /etc/ceph/cluster2.client.mirror.4.keyring && sudo chmod 0644 /etc/ceph/cluster2.client.mirror.4.keyring 2023-08-26T13:31:35.070 INFO:teuthology.orchestra.run.smithi124.stdout:creating /etc/ceph/cluster2.client.mirror.4.keyring 2023-08-26T13:31:35.082 DEBUG:teuthology.orchestra.run.smithi124:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.mirror.5 /etc/ceph/cluster2.client.mirror.5.keyring && sudo chmod 0644 /etc/ceph/cluster2.client.mirror.5.keyring 2023-08-26T13:31:35.165 INFO:teuthology.orchestra.run.smithi124.stdout:creating /etc/ceph/cluster2.client.mirror.5.keyring 2023-08-26T13:31:35.179 DEBUG:teuthology.orchestra.run.smithi124:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.mirror.6 /etc/ceph/cluster2.client.mirror.6.keyring && sudo chmod 0644 /etc/ceph/cluster2.client.mirror.6.keyring 2023-08-26T13:31:35.262 INFO:teuthology.orchestra.run.smithi124.stdout:creating /etc/ceph/cluster2.client.mirror.6.keyring 2023-08-26T13:31:35.274 INFO:tasks.ceph:Running mkfs on osd nodes... 2023-08-26T13:31:35.275 INFO:tasks.ceph:ctx.disk_config.remote_to_roles_to_dev: {Remote(name='ubuntu@smithi031.front.sepia.ceph.com'): {'cluster1.osd.0': '/dev/vg_nvme/lv_1', 'cluster1.osd.1': '/dev/vg_nvme/lv_2', 'cluster1.osd.2': '/dev/vg_nvme/lv_3'}, Remote(name='ubuntu@smithi124.front.sepia.ceph.com'): {'cluster2.osd.0': '/dev/vg_nvme/lv_1', 'cluster2.osd.1': '/dev/vg_nvme/lv_2', 'cluster2.osd.2': '/dev/vg_nvme/lv_3'}} 2023-08-26T13:31:35.275 DEBUG:teuthology.orchestra.run.smithi124:> sudo mkdir -p /var/lib/ceph/osd/cluster2-0 2023-08-26T13:31:35.341 INFO:tasks.ceph:roles_to_devs: {'cluster2.osd.0': '/dev/vg_nvme/lv_1', 'cluster2.osd.1': '/dev/vg_nvme/lv_2', 'cluster2.osd.2': '/dev/vg_nvme/lv_3'} 2023-08-26T13:31:35.341 INFO:tasks.ceph:role: cluster2.osd.0 2023-08-26T13:31:35.341 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi124.front.sepia.ceph.com 2023-08-26T13:31:35.342 DEBUG:teuthology.orchestra.run.smithi124:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2023-08-26T13:31:35.414 INFO:teuthology.orchestra.run.smithi124.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2023-08-26T13:31:35.415 INFO:teuthology.orchestra.run.smithi124.stdout: = sectsz=512 attr=2, projid32bit=1 2023-08-26T13:31:35.415 INFO:teuthology.orchestra.run.smithi124.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2023-08-26T13:31:35.415 INFO:teuthology.orchestra.run.smithi124.stdout: = reflink=1 bigtime=0 inobtcount=0 2023-08-26T13:31:35.415 INFO:teuthology.orchestra.run.smithi124.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2023-08-26T13:31:35.415 INFO:teuthology.orchestra.run.smithi124.stdout: = sunit=0 swidth=0 blks 2023-08-26T13:31:35.416 INFO:teuthology.orchestra.run.smithi124.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2023-08-26T13:31:35.416 INFO:teuthology.orchestra.run.smithi124.stdout:log =internal log bsize=4096 blocks=11443, version=2 2023-08-26T13:31:35.416 INFO:teuthology.orchestra.run.smithi124.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2023-08-26T13:31:35.416 INFO:teuthology.orchestra.run.smithi124.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2023-08-26T13:31:35.845 INFO:teuthology.orchestra.run.smithi124.stdout:Discarding blocks...Done. 2023-08-26T13:31:35.847 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi124.front.sepia.ceph.com -o noatime 2023-08-26T13:31:35.848 DEBUG:teuthology.orchestra.run.smithi124:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/cluster2-0 2023-08-26T13:31:36.034 DEBUG:teuthology.orchestra.run.smithi124:> sudo /sbin/restorecon /var/lib/ceph/osd/cluster2-0 2023-08-26T13:31:36.071 DEBUG:teuthology.orchestra.run.smithi124:> sudo mkdir -p /var/lib/ceph/osd/cluster2-1 2023-08-26T13:31:36.139 INFO:tasks.ceph:roles_to_devs: {'cluster2.osd.0': '/dev/vg_nvme/lv_1', 'cluster2.osd.1': '/dev/vg_nvme/lv_2', 'cluster2.osd.2': '/dev/vg_nvme/lv_3'} 2023-08-26T13:31:36.139 INFO:tasks.ceph:role: cluster2.osd.1 2023-08-26T13:31:36.140 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi124.front.sepia.ceph.com 2023-08-26T13:31:36.140 DEBUG:teuthology.orchestra.run.smithi124:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2023-08-26T13:31:36.214 INFO:teuthology.orchestra.run.smithi124.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2023-08-26T13:31:36.215 INFO:teuthology.orchestra.run.smithi124.stdout: = sectsz=512 attr=2, projid32bit=1 2023-08-26T13:31:36.215 INFO:teuthology.orchestra.run.smithi124.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2023-08-26T13:31:36.215 INFO:teuthology.orchestra.run.smithi124.stdout: = reflink=1 bigtime=0 inobtcount=0 2023-08-26T13:31:36.215 INFO:teuthology.orchestra.run.smithi124.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2023-08-26T13:31:36.215 INFO:teuthology.orchestra.run.smithi124.stdout: = sunit=0 swidth=0 blks 2023-08-26T13:31:36.216 INFO:teuthology.orchestra.run.smithi124.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2023-08-26T13:31:36.216 INFO:teuthology.orchestra.run.smithi124.stdout:log =internal log bsize=4096 blocks=11443, version=2 2023-08-26T13:31:36.216 INFO:teuthology.orchestra.run.smithi124.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2023-08-26T13:31:36.216 INFO:teuthology.orchestra.run.smithi124.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2023-08-26T13:31:36.653 INFO:teuthology.orchestra.run.smithi124.stdout:Discarding blocks...Done. 2023-08-26T13:31:36.654 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi124.front.sepia.ceph.com -o noatime 2023-08-26T13:31:36.655 DEBUG:teuthology.orchestra.run.smithi124:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/cluster2-1 2023-08-26T13:31:36.763 DEBUG:teuthology.orchestra.run.smithi124:> sudo /sbin/restorecon /var/lib/ceph/osd/cluster2-1 2023-08-26T13:31:36.795 DEBUG:teuthology.orchestra.run.smithi124:> sudo mkdir -p /var/lib/ceph/osd/cluster2-2 2023-08-26T13:31:36.862 INFO:tasks.ceph:roles_to_devs: {'cluster2.osd.0': '/dev/vg_nvme/lv_1', 'cluster2.osd.1': '/dev/vg_nvme/lv_2', 'cluster2.osd.2': '/dev/vg_nvme/lv_3'} 2023-08-26T13:31:36.863 INFO:tasks.ceph:role: cluster2.osd.2 2023-08-26T13:31:36.863 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi124.front.sepia.ceph.com 2023-08-26T13:31:36.864 DEBUG:teuthology.orchestra.run.smithi124:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2023-08-26T13:31:36.935 INFO:teuthology.orchestra.run.smithi124.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2023-08-26T13:31:36.936 INFO:teuthology.orchestra.run.smithi124.stdout: = sectsz=512 attr=2, projid32bit=1 2023-08-26T13:31:36.936 INFO:teuthology.orchestra.run.smithi124.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2023-08-26T13:31:36.936 INFO:teuthology.orchestra.run.smithi124.stdout: = reflink=1 bigtime=0 inobtcount=0 2023-08-26T13:31:36.936 INFO:teuthology.orchestra.run.smithi124.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2023-08-26T13:31:36.937 INFO:teuthology.orchestra.run.smithi124.stdout: = sunit=0 swidth=0 blks 2023-08-26T13:31:36.937 INFO:teuthology.orchestra.run.smithi124.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2023-08-26T13:31:36.937 INFO:teuthology.orchestra.run.smithi124.stdout:log =internal log bsize=4096 blocks=11443, version=2 2023-08-26T13:31:36.937 INFO:teuthology.orchestra.run.smithi124.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2023-08-26T13:31:36.937 INFO:teuthology.orchestra.run.smithi124.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2023-08-26T13:31:37.376 INFO:teuthology.orchestra.run.smithi124.stdout:Discarding blocks...Done. 2023-08-26T13:31:37.378 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi124.front.sepia.ceph.com -o noatime 2023-08-26T13:31:37.379 DEBUG:teuthology.orchestra.run.smithi124:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/cluster2-2 2023-08-26T13:31:37.516 DEBUG:teuthology.orchestra.run.smithi124:> sudo /sbin/restorecon /var/lib/ceph/osd/cluster2-2 2023-08-26T13:31:37.548 DEBUG:teuthology.orchestra.run.smithi124:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster cluster2 --mkfs --mkkey -i 0 --monmap /home/ubuntu/cephtest/cluster2.monmap 2023-08-26T13:31:37.651 INFO:teuthology.orchestra.run.smithi124.stderr:2023-08-26T13:31:37.649+0000 7fc40ac10640 -1 auth: error reading file: /var/lib/ceph/osd/cluster2-0/keyring: can't open /var/lib/ceph/osd/cluster2-0/keyring: (2) No such file or directory 2023-08-26T13:31:37.652 INFO:teuthology.orchestra.run.smithi124.stderr:2023-08-26T13:31:37.650+0000 7fc40ac10640 -1 created new key in keyring /var/lib/ceph/osd/cluster2-0/keyring 2023-08-26T13:31:37.652 INFO:teuthology.orchestra.run.smithi124.stderr:2023-08-26T13:31:37.650+0000 7fc40ac10640 -1 bluestore(/var/lib/ceph/osd/cluster2-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/cluster2-0/block: (2) No such file or directory 2023-08-26T13:31:37.652 INFO:teuthology.orchestra.run.smithi124.stderr:2023-08-26T13:31:37.650+0000 7fc40ac10640 -1 bluestore(/var/lib/ceph/osd/cluster2-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/cluster2-0/block: (2) No such file or directory 2023-08-26T13:31:37.652 INFO:teuthology.orchestra.run.smithi124.stderr:2023-08-26T13:31:37.650+0000 7fc40ac10640 -1 bluestore(/var/lib/ceph/osd/cluster2-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/cluster2-0/block: (2) No such file or directory 2023-08-26T13:31:37.658 INFO:teuthology.orchestra.run.smithi124.stderr:2023-08-26T13:31:37.656+0000 7fc40ac10640 -1 bluestore(/var/lib/ceph/osd/cluster2-0) _read_fsid unparsable uuid 2023-08-26T13:31:40.528 DEBUG:teuthology.orchestra.run.smithi124:> sudo chown -R ceph:ceph /var/lib/ceph/osd/cluster2-0 2023-08-26T13:31:40.573 DEBUG:teuthology.orchestra.run.smithi124:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster cluster2 --mkfs --mkkey -i 1 --monmap /home/ubuntu/cephtest/cluster2.monmap 2023-08-26T13:31:40.669 INFO:teuthology.orchestra.run.smithi124.stderr:2023-08-26T13:31:40.666+0000 7f63a77c3640 -1 auth: error reading file: /var/lib/ceph/osd/cluster2-1/keyring: can't open /var/lib/ceph/osd/cluster2-1/keyring: (2) No such file or directory 2023-08-26T13:31:40.669 INFO:teuthology.orchestra.run.smithi124.stderr:2023-08-26T13:31:40.666+0000 7f63a77c3640 -1 created new key in keyring /var/lib/ceph/osd/cluster2-1/keyring 2023-08-26T13:31:40.669 INFO:teuthology.orchestra.run.smithi124.stderr:2023-08-26T13:31:40.666+0000 7f63a77c3640 -1 bluestore(/var/lib/ceph/osd/cluster2-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/cluster2-1/block: (2) No such file or directory 2023-08-26T13:31:40.669 INFO:teuthology.orchestra.run.smithi124.stderr:2023-08-26T13:31:40.666+0000 7f63a77c3640 -1 bluestore(/var/lib/ceph/osd/cluster2-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/cluster2-1/block: (2) No such file or directory 2023-08-26T13:31:40.670 INFO:teuthology.orchestra.run.smithi124.stderr:2023-08-26T13:31:40.666+0000 7f63a77c3640 -1 bluestore(/var/lib/ceph/osd/cluster2-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/cluster2-1/block: (2) No such file or directory 2023-08-26T13:31:40.670 INFO:teuthology.orchestra.run.smithi124.stderr:2023-08-26T13:31:40.667+0000 7f63a77c3640 -1 bluestore(/var/lib/ceph/osd/cluster2-1) _read_fsid unparsable uuid 2023-08-26T13:31:43.533 DEBUG:teuthology.orchestra.run.smithi124:> sudo chown -R ceph:ceph /var/lib/ceph/osd/cluster2-1 2023-08-26T13:31:43.565 DEBUG:teuthology.orchestra.run.smithi124:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster cluster2 --mkfs --mkkey -i 2 --monmap /home/ubuntu/cephtest/cluster2.monmap 2023-08-26T13:31:43.661 INFO:teuthology.orchestra.run.smithi124.stderr:2023-08-26T13:31:43.659+0000 7f33c649a640 -1 auth: error reading file: /var/lib/ceph/osd/cluster2-2/keyring: can't open /var/lib/ceph/osd/cluster2-2/keyring: (2) No such file or directory 2023-08-26T13:31:43.661 INFO:teuthology.orchestra.run.smithi124.stderr:2023-08-26T13:31:43.659+0000 7f33c649a640 -1 created new key in keyring /var/lib/ceph/osd/cluster2-2/keyring 2023-08-26T13:31:43.662 INFO:teuthology.orchestra.run.smithi124.stderr:2023-08-26T13:31:43.659+0000 7f33c649a640 -1 bluestore(/var/lib/ceph/osd/cluster2-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/cluster2-2/block: (2) No such file or directory 2023-08-26T13:31:43.662 INFO:teuthology.orchestra.run.smithi124.stderr:2023-08-26T13:31:43.659+0000 7f33c649a640 -1 bluestore(/var/lib/ceph/osd/cluster2-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/cluster2-2/block: (2) No such file or directory 2023-08-26T13:31:43.662 INFO:teuthology.orchestra.run.smithi124.stderr:2023-08-26T13:31:43.659+0000 7f33c649a640 -1 bluestore(/var/lib/ceph/osd/cluster2-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/cluster2-2/block: (2) No such file or directory 2023-08-26T13:31:43.662 INFO:teuthology.orchestra.run.smithi124.stderr:2023-08-26T13:31:43.660+0000 7f33c649a640 -1 bluestore(/var/lib/ceph/osd/cluster2-2) _read_fsid unparsable uuid 2023-08-26T13:31:46.530 DEBUG:teuthology.orchestra.run.smithi124:> sudo chown -R ceph:ceph /var/lib/ceph/osd/cluster2-2 2023-08-26T13:31:46.562 INFO:tasks.ceph:Reading keys from all nodes... 2023-08-26T13:31:46.563 DEBUG:teuthology.orchestra.run.smithi031:> set -ex 2023-08-26T13:31:46.563 DEBUG:teuthology.orchestra.run.smithi031:> sudo dd if=/var/lib/ceph/mgr/cluster2-x/keyring of=/dev/stdout 2023-08-26T13:31:46.596 DEBUG:teuthology.orchestra.run.smithi124:> set -ex 2023-08-26T13:31:46.597 DEBUG:teuthology.orchestra.run.smithi124:> sudo dd if=/var/lib/ceph/osd/cluster2-0/keyring of=/dev/stdout 2023-08-26T13:31:46.634 DEBUG:teuthology.orchestra.run.smithi124:> set -ex 2023-08-26T13:31:46.635 DEBUG:teuthology.orchestra.run.smithi124:> sudo dd if=/var/lib/ceph/osd/cluster2-1/keyring of=/dev/stdout 2023-08-26T13:31:46.707 DEBUG:teuthology.orchestra.run.smithi124:> set -ex 2023-08-26T13:31:46.707 DEBUG:teuthology.orchestra.run.smithi124:> sudo dd if=/var/lib/ceph/osd/cluster2-2/keyring of=/dev/stdout 2023-08-26T13:31:46.779 DEBUG:teuthology.orchestra.run.smithi031:> set -ex 2023-08-26T13:31:46.779 DEBUG:teuthology.orchestra.run.smithi031:> dd if=/etc/ceph/cluster2.client.0.keyring of=/dev/stdout 2023-08-26T13:31:46.801 DEBUG:teuthology.orchestra.run.smithi124:> set -ex 2023-08-26T13:31:46.801 DEBUG:teuthology.orchestra.run.smithi124:> dd if=/etc/ceph/cluster2.client.mirror.keyring of=/dev/stdout 2023-08-26T13:31:46.839 DEBUG:teuthology.orchestra.run.smithi124:> set -ex 2023-08-26T13:31:46.839 DEBUG:teuthology.orchestra.run.smithi124:> dd if=/etc/ceph/cluster2.client.mirror.0.keyring of=/dev/stdout 2023-08-26T13:31:46.899 DEBUG:teuthology.orchestra.run.smithi124:> set -ex 2023-08-26T13:31:46.899 DEBUG:teuthology.orchestra.run.smithi124:> dd if=/etc/ceph/cluster2.client.mirror.1.keyring of=/dev/stdout 2023-08-26T13:31:46.958 DEBUG:teuthology.orchestra.run.smithi124:> set -ex 2023-08-26T13:31:46.959 DEBUG:teuthology.orchestra.run.smithi124:> dd if=/etc/ceph/cluster2.client.mirror.2.keyring of=/dev/stdout 2023-08-26T13:31:47.018 DEBUG:teuthology.orchestra.run.smithi124:> set -ex 2023-08-26T13:31:47.019 DEBUG:teuthology.orchestra.run.smithi124:> dd if=/etc/ceph/cluster2.client.mirror.3.keyring of=/dev/stdout 2023-08-26T13:31:47.078 DEBUG:teuthology.orchestra.run.smithi124:> set -ex 2023-08-26T13:31:47.079 DEBUG:teuthology.orchestra.run.smithi124:> dd if=/etc/ceph/cluster2.client.mirror.4.keyring of=/dev/stdout 2023-08-26T13:31:47.139 DEBUG:teuthology.orchestra.run.smithi124:> set -ex 2023-08-26T13:31:47.139 DEBUG:teuthology.orchestra.run.smithi124:> dd if=/etc/ceph/cluster2.client.mirror.5.keyring of=/dev/stdout 2023-08-26T13:31:47.199 DEBUG:teuthology.orchestra.run.smithi124:> set -ex 2023-08-26T13:31:47.199 DEBUG:teuthology.orchestra.run.smithi124:> dd if=/etc/ceph/cluster2.client.mirror.6.keyring of=/dev/stdout 2023-08-26T13:31:47.259 INFO:tasks.ceph:Adding keys to all mons... 2023-08-26T13:31:47.259 DEBUG:teuthology.orchestra.run.smithi124:> sudo tee -a /etc/ceph/cluster2.keyring 2023-08-26T13:31:47.329 INFO:teuthology.orchestra.run.smithi124.stdout:[mgr.x] 2023-08-26T13:31:47.329 INFO:teuthology.orchestra.run.smithi124.stdout: key = AQC2/ulkG5NLHxAAb4hOswXEoxRXEX9uB1mAdQ== 2023-08-26T13:31:47.330 INFO:teuthology.orchestra.run.smithi124.stdout:[osd.0] 2023-08-26T13:31:47.330 INFO:teuthology.orchestra.run.smithi124.stdout: key = AQC5/ulkZfHMJhAA5ezG6eWYzSy/7phLJaQFzA== 2023-08-26T13:31:47.330 INFO:teuthology.orchestra.run.smithi124.stdout:[osd.1] 2023-08-26T13:31:47.330 INFO:teuthology.orchestra.run.smithi124.stdout: key = AQC8/ulkB37QJxAAmokSyYwapR5V/4lGQK2cpA== 2023-08-26T13:31:47.330 INFO:teuthology.orchestra.run.smithi124.stdout:[osd.2] 2023-08-26T13:31:47.331 INFO:teuthology.orchestra.run.smithi124.stdout: key = AQC//ulk1tlgJxAA/Rk3+p/GNDy/CFZq/kl7Bg== 2023-08-26T13:31:47.331 INFO:teuthology.orchestra.run.smithi124.stdout:[client.0] 2023-08-26T13:31:47.331 INFO:teuthology.orchestra.run.smithi124.stdout: key = AQC2/ulkyWVCIhAAh+kNM1QwsgQw9MUTXV1+0Q== 2023-08-26T13:31:47.331 INFO:teuthology.orchestra.run.smithi124.stdout:[client.mirror] 2023-08-26T13:31:47.331 INFO:teuthology.orchestra.run.smithi124.stdout: key = AQC2/ulktySnJRAAiHChjKIPtu9WrTmhE/YdDg== 2023-08-26T13:31:47.332 INFO:teuthology.orchestra.run.smithi124.stdout:[client.mirror.0] 2023-08-26T13:31:47.332 INFO:teuthology.orchestra.run.smithi124.stdout: key = AQC2/ulkO5peKxAAGrZE/kpLtHe+0aYk6vaWxQ== 2023-08-26T13:31:47.332 INFO:teuthology.orchestra.run.smithi124.stdout:[client.mirror.1] 2023-08-26T13:31:47.332 INFO:teuthology.orchestra.run.smithi124.stdout: key = AQC2/ulkiHUQMRAA2dgkC8ya0q0Ob6ojVagBuA== 2023-08-26T13:31:47.332 INFO:teuthology.orchestra.run.smithi124.stdout:[client.mirror.2] 2023-08-26T13:31:47.333 INFO:teuthology.orchestra.run.smithi124.stdout: key = AQC2/ulkQ0bENhAARAw+S2woaj02R/6zef+OQw== 2023-08-26T13:31:47.333 INFO:teuthology.orchestra.run.smithi124.stdout:[client.mirror.3] 2023-08-26T13:31:47.333 INFO:teuthology.orchestra.run.smithi124.stdout: key = AQC2/ulk7g0QOhAA28EWIs3XNqBLjIn/Xn5e7A== 2023-08-26T13:31:47.333 INFO:teuthology.orchestra.run.smithi124.stdout:[client.mirror.4] 2023-08-26T13:31:47.333 INFO:teuthology.orchestra.run.smithi124.stdout: key = AQC3/ulkl5gqBBAA3K93XWxyE4zPeJFDvcbxNg== 2023-08-26T13:31:47.333 INFO:teuthology.orchestra.run.smithi124.stdout:[client.mirror.5] 2023-08-26T13:31:47.334 INFO:teuthology.orchestra.run.smithi124.stdout: key = AQC3/ulk4UHfCRAAp0NzmGyu8m+/21Vs+ihtEg== 2023-08-26T13:31:47.334 INFO:teuthology.orchestra.run.smithi124.stdout:[client.mirror.6] 2023-08-26T13:31:47.334 INFO:teuthology.orchestra.run.smithi124.stdout: key = AQC3/ulkEiifDxAAbYMVJx2BRutita1/5HK8EQ== 2023-08-26T13:31:47.335 DEBUG:teuthology.orchestra.run.smithi124:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster2.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2023-08-26T13:31:47.438 DEBUG:teuthology.orchestra.run.smithi124:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster2.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-08-26T13:31:47.500 DEBUG:teuthology.orchestra.run.smithi124:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster2.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-08-26T13:31:47.563 DEBUG:teuthology.orchestra.run.smithi124:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster2.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-08-26T13:31:47.626 DEBUG:teuthology.orchestra.run.smithi124:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster2.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2023-08-26T13:31:47.692 DEBUG:teuthology.orchestra.run.smithi124:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster2.keyring --name=client.mirror --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2023-08-26T13:31:47.757 DEBUG:teuthology.orchestra.run.smithi124:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster2.keyring --name=client.mirror.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2023-08-26T13:31:47.817 DEBUG:teuthology.orchestra.run.smithi124:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster2.keyring --name=client.mirror.1 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2023-08-26T13:31:47.874 DEBUG:teuthology.orchestra.run.smithi124:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster2.keyring --name=client.mirror.2 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2023-08-26T13:31:47.930 DEBUG:teuthology.orchestra.run.smithi124:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster2.keyring --name=client.mirror.3 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2023-08-26T13:31:47.986 DEBUG:teuthology.orchestra.run.smithi124:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster2.keyring --name=client.mirror.4 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2023-08-26T13:31:48.043 DEBUG:teuthology.orchestra.run.smithi124:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster2.keyring --name=client.mirror.5 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2023-08-26T13:31:48.099 DEBUG:teuthology.orchestra.run.smithi124:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster2.keyring --name=client.mirror.6 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2023-08-26T13:31:48.155 INFO:tasks.ceph:Running mkfs on mon nodes... 2023-08-26T13:31:48.156 DEBUG:teuthology.orchestra.run.smithi124:> sudo mkdir -p /var/lib/ceph/mon/cluster2-a 2023-08-26T13:31:48.182 DEBUG:teuthology.orchestra.run.smithi124:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster cluster2 --mkfs -i a --monmap /home/ubuntu/cephtest/cluster2.monmap --keyring /etc/ceph/cluster2.keyring 2023-08-26T13:31:48.279 DEBUG:teuthology.orchestra.run.smithi124:> sudo chown -R ceph:ceph /var/lib/ceph/mon/cluster2-a 2023-08-26T13:31:48.306 DEBUG:teuthology.orchestra.run.smithi124:> rm -- /home/ubuntu/cephtest/cluster2.monmap 2023-08-26T13:31:48.362 INFO:tasks.ceph:Starting mon daemons in cluster cluster2... 2023-08-26T13:31:48.363 INFO:tasks.ceph.cluster2.mon.a:Restarting daemon 2023-08-26T13:31:48.363 DEBUG:teuthology.orchestra.run.smithi124:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster cluster2 -i a 2023-08-26T13:31:48.405 INFO:tasks.ceph.cluster2.mon.a:Started 2023-08-26T13:31:48.406 INFO:tasks.ceph:Starting mgr daemons in cluster cluster2... 2023-08-26T13:31:48.406 INFO:tasks.ceph.cluster2.mgr.x:Restarting daemon 2023-08-26T13:31:48.406 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mgr -f --cluster cluster2 -i x 2023-08-26T13:31:48.409 INFO:tasks.ceph.cluster2.mgr.x:Started 2023-08-26T13:31:48.409 INFO:tasks.ceph:Setting crush tunables to default 2023-08-26T13:31:48.410 DEBUG:teuthology.orchestra.run.smithi124:> sudo ceph --cluster cluster2 osd crush tunables default 2023-08-26T13:31:48.627 INFO:teuthology.orchestra.run.smithi124.stderr:adjusted tunables profile to default 2023-08-26T13:31:48.638 INFO:tasks.ceph:check_enable_crimson: False 2023-08-26T13:31:48.638 INFO:tasks.ceph:Starting osd daemons in cluster cluster2... 2023-08-26T13:31:48.639 DEBUG:teuthology.orchestra.run.smithi124:> set -ex 2023-08-26T13:31:48.639 DEBUG:teuthology.orchestra.run.smithi124:> sudo dd if=/var/lib/ceph/osd/cluster2-0/fsid of=/dev/stdout 2023-08-26T13:31:48.649 INFO:tasks.ceph.cluster2.mgr.x.smithi031.stderr:2023-08-26T13:31:48.648+0000 7f86d50d8200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2023-08-26T13:31:48.665 DEBUG:teuthology.orchestra.run.smithi124:> set -ex 2023-08-26T13:31:48.665 DEBUG:teuthology.orchestra.run.smithi124:> sudo dd if=/var/lib/ceph/osd/cluster2-1/fsid of=/dev/stdout 2023-08-26T13:31:48.717 INFO:tasks.ceph.cluster2.mgr.x.smithi031.stderr:2023-08-26T13:31:48.716+0000 7f86d50d8200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2023-08-26T13:31:48.731 DEBUG:teuthology.orchestra.run.smithi124:> set -ex 2023-08-26T13:31:48.732 DEBUG:teuthology.orchestra.run.smithi124:> sudo dd if=/var/lib/ceph/osd/cluster2-2/fsid of=/dev/stdout 2023-08-26T13:31:48.790 INFO:tasks.ceph.cluster2.mgr.x.smithi031.stderr:2023-08-26T13:31:48.788+0000 7f86d50d8200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2023-08-26T13:31:48.798 DEBUG:teuthology.orchestra.run.smithi124:> sudo ceph --cluster cluster2 osd new 5c86f387-b043-4f1e-8c08-8595615d6fbb 0 2023-08-26T13:31:48.870 INFO:tasks.ceph.cluster2.mgr.x.smithi031.stderr:2023-08-26T13:31:48.869+0000 7f86d50d8200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2023-08-26T13:31:49.016 INFO:tasks.ceph.cluster2.mgr.x.smithi031.stderr:2023-08-26T13:31:49.015+0000 7f86d50d8200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2023-08-26T13:31:49.045 INFO:teuthology.orchestra.run.smithi124.stdout:0 2023-08-26T13:31:49.057 DEBUG:teuthology.orchestra.run.smithi124:> sudo ceph --cluster cluster2 osd new 50f99097-8058-4a53-826d-d33a240b38aa 1 2023-08-26T13:31:49.264 INFO:teuthology.orchestra.run.smithi124.stdout:1 2023-08-26T13:31:49.276 DEBUG:teuthology.orchestra.run.smithi124:> sudo ceph --cluster cluster2 osd new 27f42091-a8f4-41a2-b971-7037b8de4d70 2 2023-08-26T13:31:49.353 INFO:tasks.ceph.cluster2.mgr.x.smithi031.stderr:2023-08-26T13:31:49.352+0000 7f86d50d8200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2023-08-26T13:31:49.471 INFO:tasks.ceph.cluster2.mgr.x.smithi031.stderr:2023-08-26T13:31:49.470+0000 7f86d50d8200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2023-08-26T13:31:49.485 INFO:teuthology.orchestra.run.smithi124.stdout:2 2023-08-26T13:31:49.497 INFO:tasks.ceph.cluster2.osd.0:Restarting daemon 2023-08-26T13:31:49.498 DEBUG:teuthology.orchestra.run.smithi124:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster cluster2 -i 0 2023-08-26T13:31:49.500 INFO:tasks.ceph.cluster2.osd.0:Started 2023-08-26T13:31:49.500 INFO:tasks.ceph.cluster2.osd.1:Restarting daemon 2023-08-26T13:31:49.500 DEBUG:teuthology.orchestra.run.smithi124:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster cluster2 -i 1 2023-08-26T13:31:49.502 INFO:tasks.ceph.cluster2.osd.1:Started 2023-08-26T13:31:49.502 INFO:tasks.ceph.cluster2.osd.2:Restarting daemon 2023-08-26T13:31:49.503 DEBUG:teuthology.orchestra.run.smithi124:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster cluster2 -i 2 2023-08-26T13:31:49.504 INFO:tasks.ceph.cluster2.osd.2:Started 2023-08-26T13:31:49.505 DEBUG:teuthology.orchestra.run.smithi124:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 osd dump --format=json 2023-08-26T13:31:49.547 INFO:tasks.ceph.cluster2.mgr.x.smithi031.stderr:2023-08-26T13:31:49.545+0000 7f86d50d8200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2023-08-26T13:31:49.630 INFO:tasks.ceph.cluster2.mgr.x.smithi031.stderr:2023-08-26T13:31:49.629+0000 7f86d50d8200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2023-08-26T13:31:49.706 INFO:teuthology.orchestra.run.smithi124.stdout: 2023-08-26T13:31:49.707 INFO:teuthology.orchestra.run.smithi124.stdout:{"epoch":5,"fsid":"95df1446-13b8-484a-b4b7-3d1f11d3b887","created":"2023-08-26T13:31:48.532070+0000","modified":"2023-08-26T13:31:49.482423+0000","last_up_change":"0.000000","last_in_change":"2023-08-26T13:31:49.482423+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"5c86f387-b043-4f1e-8c08-8595615d6fbb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"50f99097-8058-4a53-826d-d33a240b38aa","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"27f42091-a8f4-41a2-b971-7037b8de4d70","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-08-26T13:31:49.718 INFO:tasks.ceph.ceph_manager.cluster2:[] 2023-08-26T13:31:49.719 INFO:tasks.ceph:Waiting for OSDs to come up 2023-08-26T13:31:50.012 INFO:tasks.ceph.cluster2.mgr.x.smithi031.stderr:2023-08-26T13:31:50.011+0000 7f86d50d8200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2023-08-26T13:31:50.021 DEBUG:teuthology.orchestra.run.smithi124:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster cluster2 osd dump --format=json 2023-08-26T13:31:50.100 INFO:tasks.ceph.cluster2.osd.0.smithi124.stderr:2023-08-26T13:31:50.097+0000 7fb6276e4640 -1 Falling back to public interface 2023-08-26T13:31:50.100 INFO:tasks.ceph.cluster2.osd.1.smithi124.stderr:2023-08-26T13:31:50.097+0000 7f2a774c6640 -1 Falling back to public interface 2023-08-26T13:31:50.101 INFO:tasks.ceph.cluster2.osd.0.smithi124.stderr:2023-08-26T13:31:50.097+0000 7fb61eaa2700 -1 PosixStack listen unable to listen on v2:0.0.0.0:6800/0: (98) Address already in use 2023-08-26T13:31:50.101 INFO:tasks.ceph.cluster2.osd.0.smithi124.stderr:2023-08-26T13:31:50.097+0000 7fb61eaa2700 -1 PosixStack listen unable to listen on v2:0.0.0.0:6801/0: (98) Address already in use 2023-08-26T13:31:50.104 INFO:tasks.ceph.cluster2.osd.2.smithi124.stderr:2023-08-26T13:31:50.102+0000 7fb7a9f91640 -1 Falling back to public interface 2023-08-26T13:31:50.208 INFO:teuthology.misc.health.smithi124.stdout: 2023-08-26T13:31:50.208 INFO:teuthology.misc.health.smithi124.stdout:{"epoch":5,"fsid":"95df1446-13b8-484a-b4b7-3d1f11d3b887","created":"2023-08-26T13:31:48.532070+0000","modified":"2023-08-26T13:31:49.482423+0000","last_up_change":"0.000000","last_in_change":"2023-08-26T13:31:49.482423+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"5c86f387-b043-4f1e-8c08-8595615d6fbb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"50f99097-8058-4a53-826d-d33a240b38aa","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"27f42091-a8f4-41a2-b971-7037b8de4d70","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-08-26T13:31:50.219 DEBUG:teuthology.misc:0 of 3 OSDs are up 2023-08-26T13:31:50.616 INFO:tasks.ceph.cluster2.mgr.x.smithi031.stderr:2023-08-26T13:31:50.615+0000 7f86d50d8200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2023-08-26T13:31:50.687 INFO:tasks.ceph.cluster2.mgr.x.smithi031.stderr:2023-08-26T13:31:50.686+0000 7f86d50d8200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2023-08-26T13:31:51.345 INFO:tasks.ceph.cluster2.mgr.x.smithi031.stderr:2023-08-26T13:31:51.343+0000 7f86d50d8200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2023-08-26T13:31:51.424 INFO:tasks.ceph.cluster2.mgr.x.smithi031.stderr:2023-08-26T13:31:51.423+0000 7f86d50d8200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2023-08-26T13:31:51.570 INFO:tasks.ceph.cluster2.mgr.x.smithi031.stderr:2023-08-26T13:31:51.568+0000 7f86d50d8200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2023-08-26T13:31:51.801 INFO:tasks.ceph.cluster2.mgr.x.smithi031.stderr:2023-08-26T13:31:51.799+0000 7f86d50d8200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2023-08-26T13:31:51.875 INFO:tasks.ceph.cluster2.mgr.x.smithi031.stderr:2023-08-26T13:31:51.874+0000 7f86d50d8200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2023-08-26T13:31:52.149 INFO:tasks.ceph.cluster2.mgr.x.smithi031.stderr:2023-08-26T13:31:52.147+0000 7f86d50d8200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2023-08-26T13:31:52.372 INFO:tasks.ceph.cluster2.mgr.x.smithi031.stderr:2023-08-26T13:31:52.370+0000 7f86d50d8200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2023-08-26T13:31:52.523 INFO:tasks.ceph.cluster2.mgr.x.smithi031.stderr:2023-08-26T13:31:52.521+0000 7f86d50d8200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2023-08-26T13:31:52.639 INFO:tasks.ceph.cluster2.mgr.x.smithi031.stderr:2023-08-26T13:31:52.638+0000 7f86d50d8200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2023-08-26T13:31:52.795 INFO:tasks.ceph.cluster2.mgr.x.smithi031.stderr:2023-08-26T13:31:52.793+0000 7f86d50d8200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2023-08-26T13:31:53.064 INFO:tasks.ceph.cluster2.mgr.x.smithi031.stderr:2023-08-26T13:31:53.063+0000 7f86d50d8200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2023-08-26T13:31:53.225 INFO:tasks.ceph.cluster2.mgr.x.smithi031.stderr:2023-08-26T13:31:53.224+0000 7f86d50d8200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2023-08-26T13:31:53.572 INFO:tasks.ceph.cluster2.osd.1.smithi124.stderr:2023-08-26T13:31:53.570+0000 7f2a774c6640 -1 osd.1 0 log_to_monitors true 2023-08-26T13:31:53.631 INFO:tasks.ceph.cluster2.osd.0.smithi124.stderr:2023-08-26T13:31:53.629+0000 7fb6276e4640 -1 osd.0 0 log_to_monitors true 2023-08-26T13:31:53.643 INFO:tasks.ceph.cluster2.osd.2.smithi124.stderr:2023-08-26T13:31:53.642+0000 7fb7a9f91640 -1 osd.2 0 log_to_monitors true 2023-08-26T13:31:53.767 INFO:tasks.ceph.cluster2.mgr.x.smithi031.stderr:2023-08-26T13:31:53.765+0000 7f86d50d8200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2023-08-26T13:31:54.787 INFO:tasks.ceph.cluster2.mgr.x.smithi031.stderr:2023-08-26T13:31:54.786+0000 7f868d12c700 -1 mgr.server handle_report got status from non-daemon mon.a 2023-08-26T13:31:56.521 DEBUG:teuthology.orchestra.run.smithi124:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster cluster2 osd dump --format=json 2023-08-26T13:31:56.999 INFO:teuthology.misc.health.smithi124.stdout: 2023-08-26T13:31:57.000 INFO:teuthology.misc.health.smithi124.stdout:{"epoch":7,"fsid":"95df1446-13b8-484a-b4b7-3d1f11d3b887","created":"2023-08-26T13:31:48.532070+0000","modified":"2023-08-26T13:31:55.496219+0000","last_up_change":"0.000000","last_in_change":"2023-08-26T13:31:49.482423+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"5c86f387-b043-4f1e-8c08-8595615d6fbb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"50f99097-8058-4a53-826d-d33a240b38aa","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"27f42091-a8f4-41a2-b971-7037b8de4d70","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-08-26T13:31:57.017 DEBUG:teuthology.misc:0 of 3 OSDs are up 2023-08-26T13:31:57.254 INFO:tasks.ceph.cluster2.osd.0.smithi124.stderr:2023-08-26T13:31:57.252+0000 7fb61b29b700 -1 osd.0 0 waiting for initial osdmap 2023-08-26T13:31:57.267 INFO:tasks.ceph.cluster2.osd.0.smithi124.stderr:2023-08-26T13:31:57.265+0000 7fb613676700 -1 osd.0 7 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-08-26T13:31:57.271 INFO:tasks.ceph.cluster2.osd.1.smithi124.stderr:2023-08-26T13:31:57.269+0000 7f2a6b07d700 -1 osd.1 0 waiting for initial osdmap 2023-08-26T13:31:57.275 INFO:tasks.ceph.cluster2.osd.2.smithi124.stderr:2023-08-26T13:31:57.274+0000 7fb79f34b700 -1 osd.2 0 waiting for initial osdmap 2023-08-26T13:31:57.276 INFO:tasks.ceph.cluster2.osd.2.smithi124.stderr:2023-08-26T13:31:57.274+0000 7fb79db48700 -1 osd.2 0 waiting for initial osdmap 2023-08-26T13:31:57.280 INFO:tasks.ceph.cluster2.osd.2.smithi124.stderr:2023-08-26T13:31:57.278+0000 7fb795f23700 -1 osd.2 7 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-08-26T13:31:57.286 INFO:tasks.ceph.cluster2.osd.1.smithi124.stderr:2023-08-26T13:31:57.285+0000 7f2a63458700 -1 osd.1 7 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-08-26T13:32:03.320 DEBUG:teuthology.orchestra.run.smithi124:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster cluster2 osd dump --format=json 2023-08-26T13:32:03.640 INFO:teuthology.misc.health.smithi124.stdout: 2023-08-26T13:32:03.641 INFO:teuthology.misc.health.smithi124.stdout:{"epoch":12,"fsid":"95df1446-13b8-484a-b4b7-3d1f11d3b887","created":"2023-08-26T13:31:48.532070+0000","modified":"2023-08-26T13:32:02.509456+0000","last_up_change":"2023-08-26T13:31:58.274522+0000","last_in_change":"2023-08-26T13:31:49.482423+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-08-26T13:31:59.825669+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"12","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":2.9900000095367432,"score_stable":2.9900000095367432,"optimal_score":0.67000001668930054,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5c86f387-b043-4f1e-8c08-8595615d6fbb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6802","nonce":4132689318},{"type":"v1","addr":"172.21.15.124:6804","nonce":4132689318}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6806","nonce":4132689318},{"type":"v1","addr":"172.21.15.124:6808","nonce":4132689318}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6814","nonce":4132689318},{"type":"v1","addr":"172.21.15.124:6815","nonce":4132689318}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6810","nonce":4132689318},{"type":"v1","addr":"172.21.15.124:6812","nonce":4132689318}]},"public_addr":"172.21.15.124:6804/4132689318","cluster_addr":"172.21.15.124:6808/4132689318","heartbeat_back_addr":"172.21.15.124:6815/4132689318","heartbeat_front_addr":"172.21.15.124:6812/4132689318","state":["exists","up"]},{"osd":1,"uuid":"50f99097-8058-4a53-826d-d33a240b38aa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":10,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6800","nonce":4079363186},{"type":"v1","addr":"172.21.15.124:6801","nonce":4079363186}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6803","nonce":4079363186},{"type":"v1","addr":"172.21.15.124:6805","nonce":4079363186}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6811","nonce":4079363186},{"type":"v1","addr":"172.21.15.124:6813","nonce":4079363186}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6807","nonce":4079363186},{"type":"v1","addr":"172.21.15.124:6809","nonce":4079363186}]},"public_addr":"172.21.15.124:6801/4079363186","cluster_addr":"172.21.15.124:6805/4079363186","heartbeat_back_addr":"172.21.15.124:6813/4079363186","heartbeat_front_addr":"172.21.15.124:6809/4079363186","state":["exists","up"]},{"osd":2,"uuid":"27f42091-a8f4-41a2-b971-7037b8de4d70","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6816","nonce":2738061324},{"type":"v1","addr":"172.21.15.124:6817","nonce":2738061324}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6818","nonce":2738061324},{"type":"v1","addr":"172.21.15.124:6819","nonce":2738061324}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6822","nonce":2738061324},{"type":"v1","addr":"172.21.15.124:6823","nonce":2738061324}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6820","nonce":2738061324},{"type":"v1","addr":"172.21.15.124:6821","nonce":2738061324}]},"public_addr":"172.21.15.124:6817/2738061324","cluster_addr":"172.21.15.124:6819/2738061324","heartbeat_back_addr":"172.21.15.124:6823/2738061324","heartbeat_front_addr":"172.21.15.124:6821/2738061324","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-08-26T13:31:54.590433+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-08-26T13:31:54.592869+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-08-26T13:31:54.661242+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-08-26T13:32:03.654 DEBUG:teuthology.misc:3 of 3 OSDs are up 2023-08-26T13:32:03.654 INFO:tasks.ceph:Creating RBD pool 2023-08-26T13:32:03.655 DEBUG:teuthology.orchestra.run.smithi124:> sudo ceph --cluster cluster2 osd pool create rbd 8 2023-08-26T13:32:04.525 INFO:teuthology.orchestra.run.smithi124.stderr:pool 'rbd' created 2023-08-26T13:32:04.538 DEBUG:teuthology.orchestra.run.smithi124:> sudo ceph --cluster cluster2 osd pool application enable rbd rbd --yes-i-really-mean-it 2023-08-26T13:32:05.528 INFO:teuthology.orchestra.run.smithi124.stderr:enabled application 'rbd' on pool 'rbd' 2023-08-26T13:32:05.540 INFO:tasks.ceph:Starting mds daemons in cluster cluster2... 2023-08-26T13:32:05.541 INFO:tasks.ceph:Waiting until cluster2 daemons up and pgs clean... 2023-08-26T13:32:05.541 INFO:tasks.ceph.ceph_manager.cluster2:waiting for mgr available 2023-08-26T13:32:05.542 DEBUG:teuthology.orchestra.run.smithi124:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 mgr dump --format=json 2023-08-26T13:32:05.542 INFO:tasks.daemonwatchdog.daemon_watchdog:watchdog starting 2023-08-26T13:32:05.917 INFO:teuthology.orchestra.run.smithi124.stdout: 2023-08-26T13:32:05.930 INFO:teuthology.orchestra.run.smithi124.stdout:{"epoch":5,"active_gid":4099,"active_name":"x","active_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6826","nonce":2922585794},{"type":"v1","addr":"172.21.15.31:6827","nonce":2922585794}]},"active_addr":"172.21.15.31:6827/2922585794","active_change":"2023-08-26T13:31:53.780110+0000","active_mgr_features":4540138322906710015,"available":true,"standbys":[],"modules":["iostat","nfs","restful"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","upmap"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:9.4.7","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:0.0.2","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"secs","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"noautoscale":{"name":"noautoscale","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"global autoscale flag","long_desc":"Option to turn on/off the autoscaler for all pools","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"drive_group_interval":{"name":"drive_group_interval","type":"float","level":"advanced","flags":0,"default_value":"300.0","min":"","max":"","enum_allowed":[],"desc":"interval in seconds between re-application of applied drive_groups","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{},"always_on_modules":{"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"quincy":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"reef":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"]},"last_failure_osd_epoch":0,"active_clients":[{"name":"devicehealth","addrvec":[{"type":"v2","addr":"172.21.15.31:0","nonce":1648727783}]},{"name":"libcephsqlite","addrvec":[{"type":"v2","addr":"172.21.15.31:0","nonce":1671888973}]},{"name":"rbd_support","addrvec":[{"type":"v2","addr":"172.21.15.31:0","nonce":3759031752}]},{"name":"volumes","addrvec":[{"type":"v2","addr":"172.21.15.31:0","nonce":306012886}]}]} 2023-08-26T13:32:05.932 INFO:tasks.ceph.ceph_manager.cluster2:mgr available! 2023-08-26T13:32:05.932 INFO:tasks.ceph.ceph_manager.cluster2:waiting for all up 2023-08-26T13:32:05.933 DEBUG:teuthology.orchestra.run.smithi124:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 osd dump --format=json 2023-08-26T13:32:06.252 INFO:teuthology.orchestra.run.smithi124.stdout: 2023-08-26T13:32:06.252 INFO:teuthology.orchestra.run.smithi124.stdout:{"epoch":14,"fsid":"95df1446-13b8-484a-b4b7-3d1f11d3b887","created":"2023-08-26T13:31:48.532070+0000","modified":"2023-08-26T13:32:05.516752+0000","last_up_change":"2023-08-26T13:31:58.274522+0000","last_in_change":"2023-08-26T13:31:49.482423+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-08-26T13:31:59.825669+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"12","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":2.9900000095367432,"score_stable":2.9900000095367432,"optimal_score":0.67000001668930054,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2023-08-26T13:32:03.988547+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"14","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5c86f387-b043-4f1e-8c08-8595615d6fbb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":13,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6802","nonce":4132689318},{"type":"v1","addr":"172.21.15.124:6804","nonce":4132689318}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6806","nonce":4132689318},{"type":"v1","addr":"172.21.15.124:6808","nonce":4132689318}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6814","nonce":4132689318},{"type":"v1","addr":"172.21.15.124:6815","nonce":4132689318}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6810","nonce":4132689318},{"type":"v1","addr":"172.21.15.124:6812","nonce":4132689318}]},"public_addr":"172.21.15.124:6804/4132689318","cluster_addr":"172.21.15.124:6808/4132689318","heartbeat_back_addr":"172.21.15.124:6815/4132689318","heartbeat_front_addr":"172.21.15.124:6812/4132689318","state":["exists","up"]},{"osd":1,"uuid":"50f99097-8058-4a53-826d-d33a240b38aa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":13,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6800","nonce":4079363186},{"type":"v1","addr":"172.21.15.124:6801","nonce":4079363186}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6803","nonce":4079363186},{"type":"v1","addr":"172.21.15.124:6805","nonce":4079363186}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6811","nonce":4079363186},{"type":"v1","addr":"172.21.15.124:6813","nonce":4079363186}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6807","nonce":4079363186},{"type":"v1","addr":"172.21.15.124:6809","nonce":4079363186}]},"public_addr":"172.21.15.124:6801/4079363186","cluster_addr":"172.21.15.124:6805/4079363186","heartbeat_back_addr":"172.21.15.124:6813/4079363186","heartbeat_front_addr":"172.21.15.124:6809/4079363186","state":["exists","up"]},{"osd":2,"uuid":"27f42091-a8f4-41a2-b971-7037b8de4d70","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":13,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6816","nonce":2738061324},{"type":"v1","addr":"172.21.15.124:6817","nonce":2738061324}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6818","nonce":2738061324},{"type":"v1","addr":"172.21.15.124:6819","nonce":2738061324}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6822","nonce":2738061324},{"type":"v1","addr":"172.21.15.124:6823","nonce":2738061324}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6820","nonce":2738061324},{"type":"v1","addr":"172.21.15.124:6821","nonce":2738061324}]},"public_addr":"172.21.15.124:6817/2738061324","cluster_addr":"172.21.15.124:6819/2738061324","heartbeat_back_addr":"172.21.15.124:6823/2738061324","heartbeat_front_addr":"172.21.15.124:6821/2738061324","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-08-26T13:31:54.590433+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-08-26T13:31:54.592869+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-08-26T13:31:54.661242+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-08-26T13:32:06.266 INFO:tasks.ceph.ceph_manager.cluster2:all up! 2023-08-26T13:32:06.266 DEBUG:teuthology.orchestra.run.smithi124:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 osd dump --format=json 2023-08-26T13:32:06.585 INFO:teuthology.orchestra.run.smithi124.stdout: 2023-08-26T13:32:06.586 INFO:teuthology.orchestra.run.smithi124.stdout:{"epoch":15,"fsid":"95df1446-13b8-484a-b4b7-3d1f11d3b887","created":"2023-08-26T13:31:48.532070+0000","modified":"2023-08-26T13:32:06.520474+0000","last_up_change":"2023-08-26T13:31:58.274522+0000","last_in_change":"2023-08-26T13:31:49.482423+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-08-26T13:31:59.825669+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"12","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":2.9900000095367432,"score_stable":2.9900000095367432,"optimal_score":0.67000001668930054,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2023-08-26T13:32:03.988547+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"15","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5c86f387-b043-4f1e-8c08-8595615d6fbb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":13,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6802","nonce":4132689318},{"type":"v1","addr":"172.21.15.124:6804","nonce":4132689318}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6806","nonce":4132689318},{"type":"v1","addr":"172.21.15.124:6808","nonce":4132689318}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6814","nonce":4132689318},{"type":"v1","addr":"172.21.15.124:6815","nonce":4132689318}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6810","nonce":4132689318},{"type":"v1","addr":"172.21.15.124:6812","nonce":4132689318}]},"public_addr":"172.21.15.124:6804/4132689318","cluster_addr":"172.21.15.124:6808/4132689318","heartbeat_back_addr":"172.21.15.124:6815/4132689318","heartbeat_front_addr":"172.21.15.124:6812/4132689318","state":["exists","up"]},{"osd":1,"uuid":"50f99097-8058-4a53-826d-d33a240b38aa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":13,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6800","nonce":4079363186},{"type":"v1","addr":"172.21.15.124:6801","nonce":4079363186}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6803","nonce":4079363186},{"type":"v1","addr":"172.21.15.124:6805","nonce":4079363186}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6811","nonce":4079363186},{"type":"v1","addr":"172.21.15.124:6813","nonce":4079363186}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6807","nonce":4079363186},{"type":"v1","addr":"172.21.15.124:6809","nonce":4079363186}]},"public_addr":"172.21.15.124:6801/4079363186","cluster_addr":"172.21.15.124:6805/4079363186","heartbeat_back_addr":"172.21.15.124:6813/4079363186","heartbeat_front_addr":"172.21.15.124:6809/4079363186","state":["exists","up"]},{"osd":2,"uuid":"27f42091-a8f4-41a2-b971-7037b8de4d70","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":13,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6816","nonce":2738061324},{"type":"v1","addr":"172.21.15.124:6817","nonce":2738061324}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6818","nonce":2738061324},{"type":"v1","addr":"172.21.15.124:6819","nonce":2738061324}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6822","nonce":2738061324},{"type":"v1","addr":"172.21.15.124:6823","nonce":2738061324}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6820","nonce":2738061324},{"type":"v1","addr":"172.21.15.124:6821","nonce":2738061324}]},"public_addr":"172.21.15.124:6817/2738061324","cluster_addr":"172.21.15.124:6819/2738061324","heartbeat_back_addr":"172.21.15.124:6823/2738061324","heartbeat_front_addr":"172.21.15.124:6821/2738061324","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-08-26T13:31:54.590433+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-08-26T13:31:54.592869+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-08-26T13:31:54.661242+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-08-26T13:32:06.599 DEBUG:teuthology.orchestra.run.smithi124:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 tell osd.0 flush_pg_stats 2023-08-26T13:32:06.600 DEBUG:teuthology.orchestra.run.smithi124:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 tell osd.1 flush_pg_stats 2023-08-26T13:32:06.600 DEBUG:teuthology.orchestra.run.smithi124:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 tell osd.2 flush_pg_stats 2023-08-26T13:32:06.764 INFO:teuthology.orchestra.run.smithi124.stdout:34359738374 2023-08-26T13:32:06.765 DEBUG:teuthology.orchestra.run.smithi124:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 osd last-stat-seq osd.1 2023-08-26T13:32:06.777 INFO:teuthology.orchestra.run.smithi124.stdout:34359738372 2023-08-26T13:32:06.778 DEBUG:teuthology.orchestra.run.smithi124:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 osd last-stat-seq osd.0 2023-08-26T13:32:06.779 INFO:teuthology.orchestra.run.smithi124.stdout:34359738373 2023-08-26T13:32:06.779 DEBUG:teuthology.orchestra.run.smithi124:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 osd last-stat-seq osd.2 2023-08-26T13:32:07.105 INFO:teuthology.orchestra.run.smithi124.stdout:34359738373 2023-08-26T13:32:07.119 INFO:tasks.ceph.ceph_manager.cluster2:need seq 34359738374 got 34359738373 for osd.1 2023-08-26T13:32:07.153 INFO:teuthology.orchestra.run.smithi124.stdout:34359738372 2023-08-26T13:32:07.161 INFO:teuthology.orchestra.run.smithi124.stdout:34359738371 2023-08-26T13:32:07.167 INFO:tasks.ceph.ceph_manager.cluster2:need seq 34359738373 got 34359738372 for osd.2 2023-08-26T13:32:07.176 INFO:tasks.ceph.ceph_manager.cluster2:need seq 34359738372 got 34359738371 for osd.0 2023-08-26T13:32:08.121 DEBUG:teuthology.orchestra.run.smithi124:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 osd last-stat-seq osd.1 2023-08-26T13:32:08.168 DEBUG:teuthology.orchestra.run.smithi124:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 osd last-stat-seq osd.2 2023-08-26T13:32:08.177 DEBUG:teuthology.orchestra.run.smithi124:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 osd last-stat-seq osd.0 2023-08-26T13:32:08.476 INFO:teuthology.orchestra.run.smithi124.stdout:34359738373 2023-08-26T13:32:08.490 INFO:tasks.ceph.ceph_manager.cluster2:need seq 34359738374 got 34359738373 for osd.1 2023-08-26T13:32:08.515 INFO:teuthology.orchestra.run.smithi124.stdout:34359738372 2023-08-26T13:32:08.529 INFO:tasks.ceph.ceph_manager.cluster2:need seq 34359738373 got 34359738372 for osd.2 2023-08-26T13:32:08.555 INFO:teuthology.orchestra.run.smithi124.stdout:34359738372 2023-08-26T13:32:08.569 INFO:tasks.ceph.ceph_manager.cluster2:need seq 34359738372 got 34359738372 for osd.0 2023-08-26T13:32:08.569 DEBUG:teuthology.parallel:result is None 2023-08-26T13:32:09.491 DEBUG:teuthology.orchestra.run.smithi124:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 osd last-stat-seq osd.1 2023-08-26T13:32:09.530 DEBUG:teuthology.orchestra.run.smithi124:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 osd last-stat-seq osd.2 2023-08-26T13:32:09.851 INFO:teuthology.orchestra.run.smithi124.stdout:34359738374 2023-08-26T13:32:09.865 INFO:tasks.ceph.ceph_manager.cluster2:need seq 34359738374 got 34359738374 for osd.1 2023-08-26T13:32:09.865 DEBUG:teuthology.parallel:result is None 2023-08-26T13:32:09.889 INFO:teuthology.orchestra.run.smithi124.stdout:34359738373 2023-08-26T13:32:09.903 INFO:tasks.ceph.ceph_manager.cluster2:need seq 34359738373 got 34359738373 for osd.2 2023-08-26T13:32:09.903 DEBUG:teuthology.parallel:result is None 2023-08-26T13:32:09.903 INFO:tasks.ceph.ceph_manager.cluster2:waiting for clean 2023-08-26T13:32:09.904 DEBUG:teuthology.orchestra.run.smithi124:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2023-08-26T13:32:10.256 INFO:teuthology.orchestra.run.smithi124.stdout: 2023-08-26T13:32:10.258 INFO:teuthology.orchestra.run.smithi124.stderr:dumped all 2023-08-26T13:32:10.269 INFO:teuthology.orchestra.run.smithi124.stdout:{"pg_ready":true,"pg_map":{"version":19,"stamp":"2023-08-26T13:32:09.786591+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":82604,"kb_used_data":1544,"kb_used_omap":5,"kb_used_meta":81018,"kb_avail":283032916,"statfs":{"total":289910292480,"available":289825705984,"internally_reserved":0,"allocated":1581056,"data_stored":1292330,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":5343,"internal_metadata":82963233},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.001384"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":11,"reported_epoch":15,"state":"active+clean","last_fresh":"2023-08-26T13:32:06.525121+0000","last_change":"2023-08-26T13:32:05.529152+0000","last_active":"2023-08-26T13:32:06.525121+0000","last_peered":"2023-08-26T13:32:06.525121+0000","last_clean":"2023-08-26T13:32:06.525121+0000","last_became_active":"2023-08-26T13:32:05.528856+0000","last_became_peered":"2023-08-26T13:32:05.528856+0000","last_unstale":"2023-08-26T13:32:06.525121+0000","last_undegraded":"2023-08-26T13:32:06.525121+0000","last_fullsized":"2023-08-26T13:32:06.525121+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-26T13:32:04.513494+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-26T13:32:04.513494+0000","last_clean_scrub_stamp":"2023-08-26T13:32:04.513494+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-27T18:24:16.009036+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":11,"reported_epoch":15,"state":"active+clean","last_fresh":"2023-08-26T13:32:06.525082+0000","last_change":"2023-08-26T13:32:05.527031+0000","last_active":"2023-08-26T13:32:06.525082+0000","last_peered":"2023-08-26T13:32:06.525082+0000","last_clean":"2023-08-26T13:32:06.525082+0000","last_became_active":"2023-08-26T13:32:05.526760+0000","last_became_peered":"2023-08-26T13:32:05.526760+0000","last_unstale":"2023-08-26T13:32:06.525082+0000","last_undegraded":"2023-08-26T13:32:06.525082+0000","last_fullsized":"2023-08-26T13:32:06.525082+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-26T13:32:04.513494+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-26T13:32:04.513494+0000","last_clean_scrub_stamp":"2023-08-26T13:32:04.513494+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-27T15:51:24.801082+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":11,"reported_epoch":15,"state":"active+clean","last_fresh":"2023-08-26T13:32:06.525208+0000","last_change":"2023-08-26T13:32:05.527186+0000","last_active":"2023-08-26T13:32:06.525208+0000","last_peered":"2023-08-26T13:32:06.525208+0000","last_clean":"2023-08-26T13:32:06.525208+0000","last_became_active":"2023-08-26T13:32:05.526955+0000","last_became_peered":"2023-08-26T13:32:05.526955+0000","last_unstale":"2023-08-26T13:32:06.525208+0000","last_undegraded":"2023-08-26T13:32:06.525208+0000","last_fullsized":"2023-08-26T13:32:06.525208+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-26T13:32:04.513494+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-26T13:32:04.513494+0000","last_clean_scrub_stamp":"2023-08-26T13:32:04.513494+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-28T00:10:43.742097+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":11,"reported_epoch":15,"state":"active+clean","last_fresh":"2023-08-26T13:32:06.775760+0000","last_change":"2023-08-26T13:32:05.529447+0000","last_active":"2023-08-26T13:32:06.775760+0000","last_peered":"2023-08-26T13:32:06.775760+0000","last_clean":"2023-08-26T13:32:06.775760+0000","last_became_active":"2023-08-26T13:32:05.529180+0000","last_became_peered":"2023-08-26T13:32:05.529180+0000","last_unstale":"2023-08-26T13:32:06.775760+0000","last_undegraded":"2023-08-26T13:32:06.775760+0000","last_fullsized":"2023-08-26T13:32:06.775760+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-26T13:32:04.513494+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-26T13:32:04.513494+0000","last_clean_scrub_stamp":"2023-08-26T13:32:04.513494+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-27T14:29:45.544431+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"11'97","reported_seq":144,"reported_epoch":15,"state":"active+clean","last_fresh":"2023-08-26T13:32:06.524929+0000","last_change":"2023-08-26T13:32:01.642164+0000","last_active":"2023-08-26T13:32:06.524929+0000","last_peered":"2023-08-26T13:32:06.524929+0000","last_clean":"2023-08-26T13:32:06.524929+0000","last_became_active":"2023-08-26T13:32:01.640918+0000","last_became_peered":"2023-08-26T13:32:01.640918+0000","last_unstale":"2023-08-26T13:32:06.524929+0000","last_undegraded":"2023-08-26T13:32:06.524929+0000","last_fullsized":"2023-08-26T13:32:06.524929+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-26T13:32:00.501196+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-26T13:32:00.501196+0000","last_clean_scrub_stamp":"2023-08-26T13:32:00.501196+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-27T15:24:24.172346+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":11,"reported_epoch":15,"state":"active+clean","last_fresh":"2023-08-26T13:32:06.525014+0000","last_change":"2023-08-26T13:32:05.530079+0000","last_active":"2023-08-26T13:32:06.525014+0000","last_peered":"2023-08-26T13:32:06.525014+0000","last_clean":"2023-08-26T13:32:06.525014+0000","last_became_active":"2023-08-26T13:32:05.529794+0000","last_became_peered":"2023-08-26T13:32:05.529794+0000","last_unstale":"2023-08-26T13:32:06.525014+0000","last_undegraded":"2023-08-26T13:32:06.525014+0000","last_fullsized":"2023-08-26T13:32:06.525014+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-26T13:32:04.513494+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-26T13:32:04.513494+0000","last_clean_scrub_stamp":"2023-08-26T13:32:04.513494+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-28T00:11:51.032901+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":11,"reported_epoch":15,"state":"active+clean","last_fresh":"2023-08-26T13:32:06.775883+0000","last_change":"2023-08-26T13:32:05.530323+0000","last_active":"2023-08-26T13:32:06.775883+0000","last_peered":"2023-08-26T13:32:06.775883+0000","last_clean":"2023-08-26T13:32:06.775883+0000","last_became_active":"2023-08-26T13:32:05.530018+0000","last_became_peered":"2023-08-26T13:32:05.530018+0000","last_unstale":"2023-08-26T13:32:06.775883+0000","last_undegraded":"2023-08-26T13:32:06.775883+0000","last_fullsized":"2023-08-26T13:32:06.775883+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-26T13:32:04.513494+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-26T13:32:04.513494+0000","last_clean_scrub_stamp":"2023-08-26T13:32:04.513494+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-27T18:27:03.411186+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":11,"reported_epoch":15,"state":"active+clean","last_fresh":"2023-08-26T13:32:06.534292+0000","last_change":"2023-08-26T13:32:05.526177+0000","last_active":"2023-08-26T13:32:06.534292+0000","last_peered":"2023-08-26T13:32:06.534292+0000","last_clean":"2023-08-26T13:32:06.534292+0000","last_became_active":"2023-08-26T13:32:05.525871+0000","last_became_peered":"2023-08-26T13:32:05.525871+0000","last_unstale":"2023-08-26T13:32:06.534292+0000","last_undegraded":"2023-08-26T13:32:06.534292+0000","last_fullsized":"2023-08-26T13:32:06.534292+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-26T13:32:04.513494+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-26T13:32:04.513494+0000","last_clean_scrub_stamp":"2023-08-26T13:32:04.513494+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-28T00:34:55.297874+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":11,"reported_epoch":15,"state":"active+clean","last_fresh":"2023-08-26T13:32:06.525011+0000","last_change":"2023-08-26T13:32:05.530106+0000","last_active":"2023-08-26T13:32:06.525011+0000","last_peered":"2023-08-26T13:32:06.525011+0000","last_clean":"2023-08-26T13:32:06.525011+0000","last_became_active":"2023-08-26T13:32:05.529825+0000","last_became_peered":"2023-08-26T13:32:05.529825+0000","last_unstale":"2023-08-26T13:32:06.525011+0000","last_undegraded":"2023-08-26T13:32:06.525011+0000","last_fullsized":"2023-08-26T13:32:06.525011+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-26T13:32:04.513494+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-26T13:32:04.513494+0000","last_clean_scrub_stamp":"2023-08-26T13:32:04.513494+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-28T00:14:45.917549+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738374,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27148,"kb_used_data":128,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344692,"statfs":{"total":96636764160,"available":96608964608,"internally_reserved":0,"allocated":131072,"data_stored":37198,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1781,"internal_metadata":27654411},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.28299999999999997}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.26400000000000001}]}]},{"osd":0,"up_from":8,"seq":34359738373,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27728,"kb_used_data":708,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344112,"statfs":{"total":96636764160,"available":96608370688,"internally_reserved":0,"allocated":724992,"data_stored":627566,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1781,"internal_metadata":27654411},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.312}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.27900000000000003}]}]},{"osd":1,"up_from":8,"seq":34359738375,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27728,"kb_used_data":708,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344112,"statfs":{"total":96636764160,"available":96608370688,"internally_reserved":0,"allocated":724992,"data_stored":627566,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1781,"internal_metadata":27654411},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.27900000000000003}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.33300000000000002}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-08-26T13:32:10.271 DEBUG:teuthology.orchestra.run.smithi124:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2023-08-26T13:32:10.585 INFO:teuthology.orchestra.run.smithi124.stdout: 2023-08-26T13:32:10.588 INFO:teuthology.orchestra.run.smithi124.stderr:dumped all 2023-08-26T13:32:10.599 INFO:teuthology.orchestra.run.smithi124.stdout:{"pg_ready":true,"pg_map":{"version":19,"stamp":"2023-08-26T13:32:09.786591+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":82604,"kb_used_data":1544,"kb_used_omap":5,"kb_used_meta":81018,"kb_avail":283032916,"statfs":{"total":289910292480,"available":289825705984,"internally_reserved":0,"allocated":1581056,"data_stored":1292330,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":5343,"internal_metadata":82963233},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.001384"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":11,"reported_epoch":15,"state":"active+clean","last_fresh":"2023-08-26T13:32:06.525121+0000","last_change":"2023-08-26T13:32:05.529152+0000","last_active":"2023-08-26T13:32:06.525121+0000","last_peered":"2023-08-26T13:32:06.525121+0000","last_clean":"2023-08-26T13:32:06.525121+0000","last_became_active":"2023-08-26T13:32:05.528856+0000","last_became_peered":"2023-08-26T13:32:05.528856+0000","last_unstale":"2023-08-26T13:32:06.525121+0000","last_undegraded":"2023-08-26T13:32:06.525121+0000","last_fullsized":"2023-08-26T13:32:06.525121+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-26T13:32:04.513494+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-26T13:32:04.513494+0000","last_clean_scrub_stamp":"2023-08-26T13:32:04.513494+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-27T18:24:16.009036+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":11,"reported_epoch":15,"state":"active+clean","last_fresh":"2023-08-26T13:32:06.525082+0000","last_change":"2023-08-26T13:32:05.527031+0000","last_active":"2023-08-26T13:32:06.525082+0000","last_peered":"2023-08-26T13:32:06.525082+0000","last_clean":"2023-08-26T13:32:06.525082+0000","last_became_active":"2023-08-26T13:32:05.526760+0000","last_became_peered":"2023-08-26T13:32:05.526760+0000","last_unstale":"2023-08-26T13:32:06.525082+0000","last_undegraded":"2023-08-26T13:32:06.525082+0000","last_fullsized":"2023-08-26T13:32:06.525082+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-26T13:32:04.513494+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-26T13:32:04.513494+0000","last_clean_scrub_stamp":"2023-08-26T13:32:04.513494+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-27T15:51:24.801082+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":11,"reported_epoch":15,"state":"active+clean","last_fresh":"2023-08-26T13:32:06.525208+0000","last_change":"2023-08-26T13:32:05.527186+0000","last_active":"2023-08-26T13:32:06.525208+0000","last_peered":"2023-08-26T13:32:06.525208+0000","last_clean":"2023-08-26T13:32:06.525208+0000","last_became_active":"2023-08-26T13:32:05.526955+0000","last_became_peered":"2023-08-26T13:32:05.526955+0000","last_unstale":"2023-08-26T13:32:06.525208+0000","last_undegraded":"2023-08-26T13:32:06.525208+0000","last_fullsized":"2023-08-26T13:32:06.525208+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-26T13:32:04.513494+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-26T13:32:04.513494+0000","last_clean_scrub_stamp":"2023-08-26T13:32:04.513494+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-28T00:10:43.742097+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":11,"reported_epoch":15,"state":"active+clean","last_fresh":"2023-08-26T13:32:06.775760+0000","last_change":"2023-08-26T13:32:05.529447+0000","last_active":"2023-08-26T13:32:06.775760+0000","last_peered":"2023-08-26T13:32:06.775760+0000","last_clean":"2023-08-26T13:32:06.775760+0000","last_became_active":"2023-08-26T13:32:05.529180+0000","last_became_peered":"2023-08-26T13:32:05.529180+0000","last_unstale":"2023-08-26T13:32:06.775760+0000","last_undegraded":"2023-08-26T13:32:06.775760+0000","last_fullsized":"2023-08-26T13:32:06.775760+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-26T13:32:04.513494+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-26T13:32:04.513494+0000","last_clean_scrub_stamp":"2023-08-26T13:32:04.513494+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-27T14:29:45.544431+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"11'97","reported_seq":144,"reported_epoch":15,"state":"active+clean","last_fresh":"2023-08-26T13:32:06.524929+0000","last_change":"2023-08-26T13:32:01.642164+0000","last_active":"2023-08-26T13:32:06.524929+0000","last_peered":"2023-08-26T13:32:06.524929+0000","last_clean":"2023-08-26T13:32:06.524929+0000","last_became_active":"2023-08-26T13:32:01.640918+0000","last_became_peered":"2023-08-26T13:32:01.640918+0000","last_unstale":"2023-08-26T13:32:06.524929+0000","last_undegraded":"2023-08-26T13:32:06.524929+0000","last_fullsized":"2023-08-26T13:32:06.524929+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-26T13:32:00.501196+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-26T13:32:00.501196+0000","last_clean_scrub_stamp":"2023-08-26T13:32:00.501196+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-27T15:24:24.172346+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":11,"reported_epoch":15,"state":"active+clean","last_fresh":"2023-08-26T13:32:06.525014+0000","last_change":"2023-08-26T13:32:05.530079+0000","last_active":"2023-08-26T13:32:06.525014+0000","last_peered":"2023-08-26T13:32:06.525014+0000","last_clean":"2023-08-26T13:32:06.525014+0000","last_became_active":"2023-08-26T13:32:05.529794+0000","last_became_peered":"2023-08-26T13:32:05.529794+0000","last_unstale":"2023-08-26T13:32:06.525014+0000","last_undegraded":"2023-08-26T13:32:06.525014+0000","last_fullsized":"2023-08-26T13:32:06.525014+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-26T13:32:04.513494+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-26T13:32:04.513494+0000","last_clean_scrub_stamp":"2023-08-26T13:32:04.513494+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-28T00:11:51.032901+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":11,"reported_epoch":15,"state":"active+clean","last_fresh":"2023-08-26T13:32:06.775883+0000","last_change":"2023-08-26T13:32:05.530323+0000","last_active":"2023-08-26T13:32:06.775883+0000","last_peered":"2023-08-26T13:32:06.775883+0000","last_clean":"2023-08-26T13:32:06.775883+0000","last_became_active":"2023-08-26T13:32:05.530018+0000","last_became_peered":"2023-08-26T13:32:05.530018+0000","last_unstale":"2023-08-26T13:32:06.775883+0000","last_undegraded":"2023-08-26T13:32:06.775883+0000","last_fullsized":"2023-08-26T13:32:06.775883+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-26T13:32:04.513494+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-26T13:32:04.513494+0000","last_clean_scrub_stamp":"2023-08-26T13:32:04.513494+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-27T18:27:03.411186+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":11,"reported_epoch":15,"state":"active+clean","last_fresh":"2023-08-26T13:32:06.534292+0000","last_change":"2023-08-26T13:32:05.526177+0000","last_active":"2023-08-26T13:32:06.534292+0000","last_peered":"2023-08-26T13:32:06.534292+0000","last_clean":"2023-08-26T13:32:06.534292+0000","last_became_active":"2023-08-26T13:32:05.525871+0000","last_became_peered":"2023-08-26T13:32:05.525871+0000","last_unstale":"2023-08-26T13:32:06.534292+0000","last_undegraded":"2023-08-26T13:32:06.534292+0000","last_fullsized":"2023-08-26T13:32:06.534292+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-26T13:32:04.513494+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-26T13:32:04.513494+0000","last_clean_scrub_stamp":"2023-08-26T13:32:04.513494+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-28T00:34:55.297874+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":11,"reported_epoch":15,"state":"active+clean","last_fresh":"2023-08-26T13:32:06.525011+0000","last_change":"2023-08-26T13:32:05.530106+0000","last_active":"2023-08-26T13:32:06.525011+0000","last_peered":"2023-08-26T13:32:06.525011+0000","last_clean":"2023-08-26T13:32:06.525011+0000","last_became_active":"2023-08-26T13:32:05.529825+0000","last_became_peered":"2023-08-26T13:32:05.529825+0000","last_unstale":"2023-08-26T13:32:06.525011+0000","last_undegraded":"2023-08-26T13:32:06.525011+0000","last_fullsized":"2023-08-26T13:32:06.525011+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-08-26T13:32:04.513494+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-08-26T13:32:04.513494+0000","last_clean_scrub_stamp":"2023-08-26T13:32:04.513494+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-08-28T00:14:45.917549+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738374,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27148,"kb_used_data":128,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344692,"statfs":{"total":96636764160,"available":96608964608,"internally_reserved":0,"allocated":131072,"data_stored":37198,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1781,"internal_metadata":27654411},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.28299999999999997}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.26400000000000001}]}]},{"osd":0,"up_from":8,"seq":34359738373,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27728,"kb_used_data":708,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344112,"statfs":{"total":96636764160,"available":96608370688,"internally_reserved":0,"allocated":724992,"data_stored":627566,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1781,"internal_metadata":27654411},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.312}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.27900000000000003}]}]},{"osd":1,"up_from":8,"seq":34359738375,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27728,"kb_used_data":708,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344112,"statfs":{"total":96636764160,"available":96608370688,"internally_reserved":0,"allocated":724992,"data_stored":627566,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1781,"internal_metadata":27654411},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.27900000000000003}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.33300000000000002}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-08-26T13:32:10.600 INFO:tasks.ceph.ceph_manager.cluster2:clean! 2023-08-26T13:32:10.600 INFO:tasks.ceph:Waiting until ceph cluster cluster2 is healthy... 2023-08-26T13:32:10.600 INFO:tasks.ceph.ceph_manager.cluster2:wait_until_healthy 2023-08-26T13:32:10.601 DEBUG:teuthology.orchestra.run.smithi124:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 health --format=json 2023-08-26T13:32:10.958 INFO:teuthology.orchestra.run.smithi124.stdout: 2023-08-26T13:32:10.958 INFO:teuthology.orchestra.run.smithi124.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2023-08-26T13:32:10.972 INFO:tasks.ceph.ceph_manager.cluster2:wait_until_healthy done 2023-08-26T13:32:10.972 INFO:teuthology.run_tasks:Running task exec... 2023-08-26T13:32:10.982 INFO:teuthology.task.exec:Executing custom commands... 2023-08-26T13:32:10.983 INFO:teuthology.task.exec:Running commands on role cluster1.client.mirror.0 host ubuntu@smithi124.front.sepia.ceph.com 2023-08-26T13:32:10.984 DEBUG:teuthology.orchestra.run.smithi124:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'sudo ceph --cluster cluster1 auth caps client.mirror mon '"'"'profile rbd-mirror-peer'"'"' osd '"'"'profile rbd'"'"'' 2023-08-26T13:32:11.391 INFO:teuthology.orchestra.run.smithi124.stdout:[client.mirror] 2023-08-26T13:32:11.391 INFO:teuthology.orchestra.run.smithi124.stdout: key = AQCQ/ulkbGRoChAA/u7HuF25JkXFTfXHFlpI5w== 2023-08-26T13:32:11.392 INFO:teuthology.orchestra.run.smithi124.stdout: caps mon = "profile rbd-mirror-peer" 2023-08-26T13:32:11.392 INFO:teuthology.orchestra.run.smithi124.stdout: caps osd = "profile rbd" 2023-08-26T13:32:11.394 INFO:teuthology.orchestra.run.smithi124.stderr:updated caps for client.mirror 2023-08-26T13:32:11.406 DEBUG:teuthology.orchestra.run.smithi124:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'sudo ceph --cluster cluster1 auth caps client.mirror.0 mon '"'"'profile rbd-mirror'"'"' osd '"'"'profile rbd'"'"'' 2023-08-26T13:32:11.815 INFO:teuthology.orchestra.run.smithi124.stdout:[client.mirror.0] 2023-08-26T13:32:11.815 INFO:teuthology.orchestra.run.smithi124.stdout: key = AQCQ/ulk+l0MDhAAKVawsGzeCVjS5ToF1AeIeg== 2023-08-26T13:32:11.815 INFO:teuthology.orchestra.run.smithi124.stdout: caps mon = "profile rbd-mirror" 2023-08-26T13:32:11.816 INFO:teuthology.orchestra.run.smithi124.stdout: caps osd = "profile rbd" 2023-08-26T13:32:11.816 INFO:teuthology.orchestra.run.smithi124.stderr:updated caps for client.mirror.0 2023-08-26T13:32:11.828 DEBUG:teuthology.orchestra.run.smithi124:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'sudo ceph --cluster cluster1 auth caps client.mirror.1 mon '"'"'profile rbd-mirror'"'"' osd '"'"'profile rbd'"'"'' 2023-08-26T13:32:12.237 INFO:teuthology.orchestra.run.smithi124.stdout:[client.mirror.1] 2023-08-26T13:32:12.237 INFO:teuthology.orchestra.run.smithi124.stdout: key = AQCQ/ulkdrVtERAAysNg9WClxAVKE29CrJahtw== 2023-08-26T13:32:12.238 INFO:teuthology.orchestra.run.smithi124.stdout: caps mon = "profile rbd-mirror" 2023-08-26T13:32:12.238 INFO:teuthology.orchestra.run.smithi124.stdout: caps osd = "profile rbd" 2023-08-26T13:32:12.239 INFO:teuthology.orchestra.run.smithi124.stderr:updated caps for client.mirror.1 2023-08-26T13:32:12.251 DEBUG:teuthology.orchestra.run.smithi124:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'sudo ceph --cluster cluster1 auth caps client.mirror.2 mon '"'"'profile rbd-mirror'"'"' osd '"'"'profile rbd'"'"'' 2023-08-26T13:32:12.665 INFO:teuthology.orchestra.run.smithi124.stdout:[client.mirror.2] 2023-08-26T13:32:12.665 INFO:teuthology.orchestra.run.smithi124.stdout: key = AQCQ/ulke08tFxAAsLvyyB7D2xHwyH/4zjNTBg== 2023-08-26T13:32:12.665 INFO:teuthology.orchestra.run.smithi124.stdout: caps mon = "profile rbd-mirror" 2023-08-26T13:32:12.665 INFO:teuthology.orchestra.run.smithi124.stdout: caps osd = "profile rbd" 2023-08-26T13:32:12.667 INFO:teuthology.orchestra.run.smithi124.stderr:updated caps for client.mirror.2 2023-08-26T13:32:12.679 DEBUG:teuthology.orchestra.run.smithi124:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'sudo ceph --cluster cluster1 auth caps client.mirror.3 mon '"'"'profile rbd-mirror'"'"' osd '"'"'profile rbd'"'"'' 2023-08-26T13:32:13.088 INFO:teuthology.orchestra.run.smithi124.stdout:[client.mirror.3] 2023-08-26T13:32:13.088 INFO:teuthology.orchestra.run.smithi124.stdout: key = AQCQ/ulkOQjnHBAAXyvouR+GF/42kzqtzDV7OQ== 2023-08-26T13:32:13.088 INFO:teuthology.orchestra.run.smithi124.stdout: caps mon = "profile rbd-mirror" 2023-08-26T13:32:13.088 INFO:teuthology.orchestra.run.smithi124.stdout: caps osd = "profile rbd" 2023-08-26T13:32:13.090 INFO:teuthology.orchestra.run.smithi124.stderr:updated caps for client.mirror.3 2023-08-26T13:32:13.102 DEBUG:teuthology.orchestra.run.smithi124:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'sudo ceph --cluster cluster2 auth caps client.mirror mon '"'"'profile rbd-mirror-peer'"'"' osd '"'"'profile rbd'"'"'' 2023-08-26T13:32:13.509 INFO:teuthology.orchestra.run.smithi124.stdout:[client.mirror] 2023-08-26T13:32:13.509 INFO:teuthology.orchestra.run.smithi124.stdout: key = AQC2/ulktySnJRAAiHChjKIPtu9WrTmhE/YdDg== 2023-08-26T13:32:13.509 INFO:teuthology.orchestra.run.smithi124.stdout: caps mon = "profile rbd-mirror-peer" 2023-08-26T13:32:13.510 INFO:teuthology.orchestra.run.smithi124.stdout: caps osd = "profile rbd" 2023-08-26T13:32:13.511 INFO:teuthology.orchestra.run.smithi124.stderr:updated caps for client.mirror 2023-08-26T13:32:13.523 DEBUG:teuthology.orchestra.run.smithi124:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'sudo ceph --cluster cluster2 auth caps client.mirror.0 mon '"'"'profile rbd-mirror'"'"' osd '"'"'profile rbd'"'"'' 2023-08-26T13:32:13.929 INFO:teuthology.orchestra.run.smithi124.stdout:[client.mirror.0] 2023-08-26T13:32:13.930 INFO:teuthology.orchestra.run.smithi124.stdout: key = AQC2/ulkO5peKxAAGrZE/kpLtHe+0aYk6vaWxQ== 2023-08-26T13:32:13.930 INFO:teuthology.orchestra.run.smithi124.stdout: caps mon = "profile rbd-mirror" 2023-08-26T13:32:13.930 INFO:teuthology.orchestra.run.smithi124.stdout: caps osd = "profile rbd" 2023-08-26T13:32:13.932 INFO:teuthology.orchestra.run.smithi124.stderr:updated caps for client.mirror.0 2023-08-26T13:32:13.944 DEBUG:teuthology.orchestra.run.smithi124:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'sudo ceph --cluster cluster2 auth caps client.mirror.1 mon '"'"'profile rbd-mirror'"'"' osd '"'"'profile rbd'"'"'' 2023-08-26T13:32:14.353 INFO:teuthology.orchestra.run.smithi124.stdout:[client.mirror.1] 2023-08-26T13:32:14.353 INFO:teuthology.orchestra.run.smithi124.stdout: key = AQC2/ulkiHUQMRAA2dgkC8ya0q0Ob6ojVagBuA== 2023-08-26T13:32:14.353 INFO:teuthology.orchestra.run.smithi124.stdout: caps mon = "profile rbd-mirror" 2023-08-26T13:32:14.354 INFO:teuthology.orchestra.run.smithi124.stdout: caps osd = "profile rbd" 2023-08-26T13:32:14.355 INFO:teuthology.orchestra.run.smithi124.stderr:updated caps for client.mirror.1 2023-08-26T13:32:14.367 DEBUG:teuthology.orchestra.run.smithi124:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'sudo ceph --cluster cluster2 auth caps client.mirror.2 mon '"'"'profile rbd-mirror'"'"' osd '"'"'profile rbd'"'"'' 2023-08-26T13:32:14.775 INFO:teuthology.orchestra.run.smithi124.stdout:[client.mirror.2] 2023-08-26T13:32:14.775 INFO:teuthology.orchestra.run.smithi124.stdout: key = AQC2/ulkQ0bENhAARAw+S2woaj02R/6zef+OQw== 2023-08-26T13:32:14.775 INFO:teuthology.orchestra.run.smithi124.stdout: caps mon = "profile rbd-mirror" 2023-08-26T13:32:14.775 INFO:teuthology.orchestra.run.smithi124.stdout: caps osd = "profile rbd" 2023-08-26T13:32:14.777 INFO:teuthology.orchestra.run.smithi124.stderr:updated caps for client.mirror.2 2023-08-26T13:32:14.789 DEBUG:teuthology.orchestra.run.smithi124:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'sudo ceph --cluster cluster2 auth caps client.mirror.3 mon '"'"'profile rbd-mirror'"'"' osd '"'"'profile rbd'"'"'' 2023-08-26T13:32:15.197 INFO:teuthology.orchestra.run.smithi124.stdout:[client.mirror.3] 2023-08-26T13:32:15.198 INFO:teuthology.orchestra.run.smithi124.stdout: key = AQC2/ulk7g0QOhAA28EWIs3XNqBLjIn/Xn5e7A== 2023-08-26T13:32:15.198 INFO:teuthology.orchestra.run.smithi124.stdout: caps mon = "profile rbd-mirror" 2023-08-26T13:32:15.199 INFO:teuthology.orchestra.run.smithi124.stdout: caps osd = "profile rbd" 2023-08-26T13:32:15.200 INFO:teuthology.orchestra.run.smithi124.stderr:updated caps for client.mirror.3 2023-08-26T13:32:15.212 INFO:teuthology.run_tasks:Running task rbd-mirror... 2023-08-26T13:32:15.223 INFO:tasks.rbd_mirror.cluster1.client.mirror.0:Restarting daemon 2023-08-26T13:32:15.224 DEBUG:teuthology.orchestra.run.smithi124:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.0 2023-08-26T13:32:15.226 INFO:tasks.rbd_mirror.cluster1.client.mirror.0:Started 2023-08-26T13:32:15.226 INFO:teuthology.run_tasks:Running task rbd-mirror... 2023-08-26T13:32:15.247 INFO:tasks.rbd_mirror.cluster1.client.mirror.1:Restarting daemon 2023-08-26T13:32:15.247 DEBUG:teuthology.orchestra.run.smithi124:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.1 2023-08-26T13:32:15.250 INFO:tasks.rbd_mirror.cluster1.client.mirror.1:Started 2023-08-26T13:32:15.250 INFO:teuthology.run_tasks:Running task rbd-mirror... 2023-08-26T13:32:15.262 INFO:tasks.rbd_mirror.cluster1.client.mirror.2:Restarting daemon 2023-08-26T13:32:15.263 DEBUG:teuthology.orchestra.run.smithi124:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.2 2023-08-26T13:32:15.265 INFO:tasks.rbd_mirror.cluster1.client.mirror.2:Started 2023-08-26T13:32:15.265 INFO:teuthology.run_tasks:Running task rbd-mirror... 2023-08-26T13:32:15.275 INFO:tasks.rbd_mirror.cluster1.client.mirror.3:Restarting daemon 2023-08-26T13:32:15.275 DEBUG:teuthology.orchestra.run.smithi124:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.3 2023-08-26T13:32:15.278 INFO:tasks.rbd_mirror.cluster1.client.mirror.3:Started 2023-08-26T13:32:15.278 INFO:teuthology.run_tasks:Running task rbd-mirror... 2023-08-26T13:32:15.294 INFO:tasks.rbd_mirror.cluster2.client.mirror.0:Restarting daemon 2023-08-26T13:32:15.295 DEBUG:teuthology.orchestra.run.smithi124:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.0 2023-08-26T13:32:15.297 INFO:tasks.rbd_mirror.cluster2.client.mirror.0:Started 2023-08-26T13:32:15.297 INFO:teuthology.run_tasks:Running task rbd-mirror... 2023-08-26T13:32:15.310 INFO:tasks.rbd_mirror.cluster2.client.mirror.1:Restarting daemon 2023-08-26T13:32:15.311 DEBUG:teuthology.orchestra.run.smithi124:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.1 2023-08-26T13:32:15.313 INFO:tasks.rbd_mirror.cluster2.client.mirror.1:Started 2023-08-26T13:32:15.314 INFO:teuthology.run_tasks:Running task rbd-mirror... 2023-08-26T13:32:15.326 INFO:tasks.rbd_mirror.cluster2.client.mirror.2:Restarting daemon 2023-08-26T13:32:15.327 DEBUG:teuthology.orchestra.run.smithi124:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.2 2023-08-26T13:32:15.332 INFO:tasks.rbd_mirror.cluster2.client.mirror.2:Started 2023-08-26T13:32:15.332 INFO:teuthology.run_tasks:Running task rbd-mirror... 2023-08-26T13:32:15.344 INFO:tasks.rbd_mirror.cluster2.client.mirror.3:Restarting daemon 2023-08-26T13:32:15.344 DEBUG:teuthology.orchestra.run.smithi124:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.3 2023-08-26T13:32:15.347 INFO:tasks.rbd_mirror.cluster2.client.mirror.3:Started 2023-08-26T13:32:15.347 INFO:teuthology.run_tasks:Running task rbd-mirror-thrash... 2023-08-26T13:32:15.364 INFO:tasks.rbd_mirror_thrash:rbd_mirror_thrash using random seed: 1693056735 2023-08-26T13:32:15.365 DEBUG:tasks.rbd_mirror_thrash:Yielding 2023-08-26T13:32:15.365 INFO:teuthology.run_tasks:Running task rbd-mirror-thrash... 2023-08-26T13:32:15.378 INFO:tasks.rbd_mirror_thrash:rbd_mirror_thrash using random seed: 1693056735 2023-08-26T13:32:15.379 DEBUG:tasks.rbd_mirror_thrash:Yielding 2023-08-26T13:32:15.379 INFO:teuthology.run_tasks:Running task workunit... 2023-08-26T13:32:15.442 INFO:tasks.workunit:Pulling workunits from ref e7eb9853fddaf661c68f207346131a7e00d8ed2f 2023-08-26T13:32:15.442 INFO:tasks.workunit:Making a separate scratch dir for every client... 2023-08-26T13:32:15.442 DEBUG:teuthology.orchestra.run.smithi124:> stat -- /home/ubuntu/cephtest/mnt.cluster1.mirror 2023-08-26T13:32:15.443 INFO:tasks.rbd_mirror_thrash:starting thrash for cluster cluster1 2023-08-26T13:32:15.443 INFO:tasks.rbd_mirror_thrash:waiting for 90 secs before thrashing 2023-08-26T13:32:15.444 INFO:tasks.rbd_mirror_thrash:starting thrash for cluster cluster2 2023-08-26T13:32:15.444 INFO:tasks.rbd_mirror_thrash:waiting for 101 secs before thrashing 2023-08-26T13:32:15.460 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-08-26T13:32:15.461 INFO:teuthology.orchestra.run.smithi124.stderr:stat: cannot statx '/home/ubuntu/cephtest/mnt.cluster1.mirror': No such file or directory 2023-08-26T13:32:15.461 DEBUG:teuthology.orchestra.run.smithi124:> mkdir -- /home/ubuntu/cephtest/mnt.cluster1.mirror 2023-08-26T13:32:15.517 INFO:tasks.workunit:Created dir /home/ubuntu/cephtest/mnt.cluster1.mirror 2023-08-26T13:32:15.517 DEBUG:teuthology.orchestra.run.smithi124:> cd -- /home/ubuntu/cephtest/mnt.cluster1.mirror && mkdir -- client.mirror 2023-08-26T13:32:15.573 INFO:tasks.workunit:timeout=6h 2023-08-26T13:32:15.574 INFO:tasks.workunit:cleanup=True 2023-08-26T13:32:15.575 DEBUG:teuthology.orchestra.run.smithi124:> rm -rf /home/ubuntu/cephtest/clone.cluster1.client.mirror && git clone https://github.com/ceph/ceph-ci.git /home/ubuntu/cephtest/clone.cluster1.client.mirror && cd /home/ubuntu/cephtest/clone.cluster1.client.mirror && git checkout e7eb9853fddaf661c68f207346131a7e00d8ed2f 2023-08-26T13:32:15.630 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:Cloning into '/home/ubuntu/cephtest/clone.cluster1.client.mirror'... 2023-08-26T13:32:55.019 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:Note: switching to 'e7eb9853fddaf661c68f207346131a7e00d8ed2f'. 2023-08-26T13:32:55.020 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr: 2023-08-26T13:32:55.020 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:You are in 'detached HEAD' state. You can look around, make experimental 2023-08-26T13:32:55.020 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:changes and commit them, and you can discard any commits you make in this 2023-08-26T13:32:55.020 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:state without impacting any branches by switching back to a branch. 2023-08-26T13:32:55.021 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr: 2023-08-26T13:32:55.021 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:If you want to create a new branch to retain commits you create, you may 2023-08-26T13:32:55.021 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:do so (now or later) by using -c with the switch command. Example: 2023-08-26T13:32:55.021 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr: 2023-08-26T13:32:55.021 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr: git switch -c 2023-08-26T13:32:55.022 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr: 2023-08-26T13:32:55.022 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:Or undo this operation with: 2023-08-26T13:32:55.022 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr: 2023-08-26T13:32:55.022 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr: git switch - 2023-08-26T13:32:55.022 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr: 2023-08-26T13:32:55.022 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:Turn off this advice by setting config variable advice.detachedHead to false 2023-08-26T13:32:55.023 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr: 2023-08-26T13:32:55.023 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:HEAD is now at e7eb9853fdd Merge pull request #53141 from rhcs-dashboard/fix-host-e2e 2023-08-26T13:32:55.024 DEBUG:teuthology.orchestra.run.smithi124:> cd -- /home/ubuntu/cephtest/clone.cluster1.client.mirror/qa/workunits && if test -e Makefile ; then make ; fi && find -executable -type f -printf '%P\0' >/home/ubuntu/cephtest/workunits.list.cluster1.client.mirror 2023-08-26T13:32:55.087 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:for d in direct_io fs ; do ( cd $d ; make all ) ; done 2023-08-26T13:32:55.089 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:make[1]: Entering directory '/home/ubuntu/cephtest/clone.cluster1.client.mirror/qa/workunits/direct_io' 2023-08-26T13:32:55.089 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:cc -Wall -Wextra -D_GNU_SOURCE direct_io_test.c -o direct_io_test 2023-08-26T13:32:55.669 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_sync_io.c -o test_sync_io 2023-08-26T13:32:55.737 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_short_dio_read.c -o test_short_dio_read 2023-08-26T13:32:55.774 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:make[1]: Leaving directory '/home/ubuntu/cephtest/clone.cluster1.client.mirror/qa/workunits/direct_io' 2023-08-26T13:32:55.775 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:make[1]: Entering directory '/home/ubuntu/cephtest/clone.cluster1.client.mirror/qa/workunits/fs' 2023-08-26T13:32:55.776 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_o_trunc.c -o test_o_trunc 2023-08-26T13:32:55.811 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:make[1]: Leaving directory '/home/ubuntu/cephtest/clone.cluster1.client.mirror/qa/workunits/fs' 2023-08-26T13:32:55.814 DEBUG:teuthology.orchestra.run.smithi124:> set -ex 2023-08-26T13:32:55.815 DEBUG:teuthology.orchestra.run.smithi124:> dd if=/home/ubuntu/cephtest/workunits.list.cluster1.client.mirror of=/dev/stdout 2023-08-26T13:32:55.872 INFO:tasks.workunit:Running workunits matching rbd/rbd_mirror_stress.sh on cluster1.client.mirror... 2023-08-26T13:32:55.873 INFO:tasks.workunit:Running workunit rbd/rbd_mirror_stress.sh... 2023-08-26T13:32:55.873 DEBUG:teuthology.orchestra.run.smithi124:workunit test rbd/rbd_mirror_stress.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.cluster1.mirror/client.mirror/tmp && cd -- /home/ubuntu/cephtest/mnt.cluster1.mirror/client.mirror/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=e7eb9853fddaf661c68f207346131a7e00d8ed2f TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster cluster1" CEPH_ID="mirror" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.cluster1.client.mirror CEPH_ROOT=/home/ubuntu/cephtest/clone.cluster1.client.mirror CEPH_MNT=/home/ubuntu/cephtest/mnt.cluster1.mirror CEPH_ARGS='' MIRROR_IMAGE_MODE=snapshot MIRROR_POOL_MODE=image RBD_IMAGE_FEATURES=layering,exclusive-lock RBD_MIRROR_INSTANCES=4 RBD_MIRROR_USE_EXISTING_CLUSTER=1 RBD_MIRROR_USE_RBD_MIRROR=1 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 6h /home/ubuntu/cephtest/clone.cluster1.client.mirror/qa/workunits/rbd/rbd_mirror_stress.sh 2023-08-26T13:32:55.932 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ IMAGE_COUNT=50 2023-08-26T13:32:55.932 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ export LOCKDEP=0 2023-08-26T13:32:55.933 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ LOCKDEP=0 2023-08-26T13:32:55.933 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ dirname /home/ubuntu/cephtest/clone.cluster1.client.mirror/qa/workunits/rbd/rbd_mirror_stress.sh 2023-08-26T13:32:55.935 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ . /home/ubuntu/cephtest/clone.cluster1.client.mirror/qa/workunits/rbd/rbd_mirror_helpers.sh 2023-08-26T13:32:55.935 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ type xmlstarlet 2023-08-26T13:32:55.935 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ XMLSTARLET=xmlstarlet 2023-08-26T13:32:55.935 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ RBD_MIRROR_INSTANCES=4 2023-08-26T13:32:55.936 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ CLUSTER1=cluster1 2023-08-26T13:32:55.936 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ CLUSTER2=cluster2 2023-08-26T13:32:55.936 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ PEER_CLUSTER_SUFFIX= 2023-08-26T13:32:55.936 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ POOL=mirror 2023-08-26T13:32:55.936 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ PARENT_POOL=mirror_parent 2023-08-26T13:32:55.936 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ NS1=ns1 2023-08-26T13:32:55.937 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ NS2=ns2 2023-08-26T13:32:55.937 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ TEMPDIR= 2023-08-26T13:32:55.937 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ CEPH_ID=mirror 2023-08-26T13:32:55.937 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ RBD_IMAGE_FEATURES=layering,exclusive-lock 2023-08-26T13:32:55.937 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ MIRROR_USER_ID_PREFIX=mirror. 2023-08-26T13:32:55.938 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ MIRROR_POOL_MODE=image 2023-08-26T13:32:55.938 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ MIRROR_IMAGE_MODE=snapshot 2023-08-26T13:32:55.938 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ export 'CEPH_ARGS=--id mirror' 2023-08-26T13:32:55.938 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ CEPH_ARGS='--id mirror' 2023-08-26T13:32:55.938 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ LAST_MIRROR_INSTANCE=3 2023-08-26T13:32:55.939 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++++ dirname /home/ubuntu/cephtest/clone.cluster1.client.mirror/qa/workunits/rbd/rbd_mirror_stress.sh 2023-08-26T13:32:55.939 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ readlink -f /home/ubuntu/cephtest/clone.cluster1.client.mirror/qa/workunits/rbd/../../../src 2023-08-26T13:32:55.941 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ CEPH_ROOT=/home/ubuntu/cephtest/clone.cluster1.client.mirror/src 2023-08-26T13:32:55.941 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ CEPH_BIN=. 2023-08-26T13:32:55.941 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ CEPH_SRC=. 2023-08-26T13:32:55.941 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ '[' -e CMakeCache.txt ']' 2023-08-26T13:32:55.941 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ '[' 0 -gt 0 ']' 2023-08-26T13:32:55.942 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ setup 2023-08-26T13:32:55.942 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local c 2023-08-26T13:32:55.942 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ trap 'cleanup $?' INT TERM EXIT 2023-08-26T13:32:55.942 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ setup_tempdir 2023-08-26T13:32:55.942 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ '[' -n '' ']' 2023-08-26T13:32:55.942 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ mktemp -d 2023-08-26T13:32:55.943 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ TEMPDIR=/tmp/tmp.TfDn6BcQ3N 2023-08-26T13:32:55.943 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ '[' -z 1 ']' 2023-08-26T13:32:55.943 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ setup_pools cluster1 cluster2 2023-08-26T13:32:55.943 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster1 2023-08-26T13:32:55.943 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local remote_cluster=cluster2 2023-08-26T13:32:55.944 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local mon_map_file 2023-08-26T13:32:55.944 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local mon_addr 2023-08-26T13:32:55.944 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local admin_key_file 2023-08-26T13:32:55.944 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local uuid 2023-08-26T13:32:55.944 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ CEPH_ARGS= 2023-08-26T13:32:55.945 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ ceph --cluster cluster1 osd pool create mirror 64 64 2023-08-26T13:32:57.247 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:pool 'mirror' already exists 2023-08-26T13:32:57.256 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ CEPH_ARGS= 2023-08-26T13:32:57.256 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ ceph --cluster cluster1 osd pool create mirror_parent 64 64 2023-08-26T13:32:58.249 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:pool 'mirror_parent' already exists 2023-08-26T13:32:58.259 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ CEPH_ARGS= 2023-08-26T13:32:58.259 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rbd --cluster cluster1 pool init mirror 2023-08-26T13:33:01.150 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ CEPH_ARGS= 2023-08-26T13:33:01.151 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rbd --cluster cluster1 pool init mirror_parent 2023-08-26T13:33:04.165 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ '[' -n '' ']' 2023-08-26T13:33:04.165 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ CEPH_ARGS= 2023-08-26T13:33:04.166 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rbd --cluster cluster1 mirror pool enable --site-name cluster1 mirror image 2023-08-26T13:33:04.211 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rbd --cluster cluster1 mirror pool enable mirror_parent image 2023-08-26T13:33:04.252 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rbd --cluster cluster1 namespace create mirror/ns1 2023-08-26T13:33:04.289 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rbd --cluster cluster1 namespace create mirror/ns2 2023-08-26T13:33:04.325 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rbd --cluster cluster1 mirror pool enable mirror/ns1 image 2023-08-26T13:33:04.363 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rbd --cluster cluster1 mirror pool enable mirror/ns2 image 2023-08-26T13:33:04.402 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ '[' -z ']' 2023-08-26T13:33:04.403 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ '[' -z ']' 2023-08-26T13:33:04.403 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ peer_add cluster1 mirror cluster2 2023-08-26T13:33:04.403 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster1 2023-08-26T13:33:04.403 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ shift 2023-08-26T13:33:04.403 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T13:33:04.403 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ shift 2023-08-26T13:33:04.404 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local client_cluster=cluster2 2023-08-26T13:33:04.404 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ shift 2023-08-26T13:33:04.404 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local remote_cluster=cluster2 2023-08-26T13:33:04.404 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local uuid_var_name 2023-08-26T13:33:04.404 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ '[' -n '' ']' 2023-08-26T13:33:04.405 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local error_code 2023-08-26T13:33:04.405 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local peer_uuid 2023-08-26T13:33:04.405 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ for s in 1 2 4 8 16 32 2023-08-26T13:33:04.405 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ set +e 2023-08-26T13:33:04.405 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ rbd --cluster cluster1 mirror pool peer add mirror cluster2 2023-08-26T13:33:04.440 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ peer_uuid=0ac69477-c0d3-495e-b791-aec031f1f9dc 2023-08-26T13:33:04.440 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ error_code=0 2023-08-26T13:33:04.440 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ set -e 2023-08-26T13:33:04.441 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ '[' 0 -eq 17 ']' 2023-08-26T13:33:04.441 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ test 0 -eq 0 2023-08-26T13:33:04.441 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ '[' -n '' ']' 2023-08-26T13:33:04.441 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ return 0 2023-08-26T13:33:04.441 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ peer_add cluster1 mirror_parent cluster2 2023-08-26T13:33:04.442 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster1 2023-08-26T13:33:04.442 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ shift 2023-08-26T13:33:04.442 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror_parent 2023-08-26T13:33:04.442 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ shift 2023-08-26T13:33:04.442 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local client_cluster=cluster2 2023-08-26T13:33:04.442 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ shift 2023-08-26T13:33:04.443 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local remote_cluster=cluster2 2023-08-26T13:33:04.443 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local uuid_var_name 2023-08-26T13:33:04.443 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ '[' -n '' ']' 2023-08-26T13:33:04.443 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local error_code 2023-08-26T13:33:04.443 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local peer_uuid 2023-08-26T13:33:04.444 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ for s in 1 2 4 8 16 32 2023-08-26T13:33:04.444 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ set +e 2023-08-26T13:33:04.444 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ rbd --cluster cluster1 mirror pool peer add mirror_parent cluster2 2023-08-26T13:33:04.476 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ peer_uuid=111b7cf3-fbb0-4f72-be72-feab89995cce 2023-08-26T13:33:04.477 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ error_code=0 2023-08-26T13:33:04.477 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ set -e 2023-08-26T13:33:04.477 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ '[' 0 -eq 17 ']' 2023-08-26T13:33:04.477 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ test 0 -eq 0 2023-08-26T13:33:04.477 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ '[' -n '' ']' 2023-08-26T13:33:04.477 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ return 0 2023-08-26T13:33:04.478 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ setup_pools cluster2 cluster1 2023-08-26T13:33:04.478 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster2 2023-08-26T13:33:04.478 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local remote_cluster=cluster1 2023-08-26T13:33:04.478 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local mon_map_file 2023-08-26T13:33:04.478 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local mon_addr 2023-08-26T13:33:04.479 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local admin_key_file 2023-08-26T13:33:04.479 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local uuid 2023-08-26T13:33:04.479 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ CEPH_ARGS= 2023-08-26T13:33:04.479 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ ceph --cluster cluster2 osd pool create mirror 64 64 2023-08-26T13:33:05.644 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:pool 'mirror' already exists 2023-08-26T13:33:05.654 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ CEPH_ARGS= 2023-08-26T13:33:05.654 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ ceph --cluster cluster2 osd pool create mirror_parent 64 64 2023-08-26T13:33:06.670 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:pool 'mirror_parent' already exists 2023-08-26T13:33:06.679 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ CEPH_ARGS= 2023-08-26T13:33:06.680 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rbd --cluster cluster2 pool init mirror 2023-08-26T13:33:09.548 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ CEPH_ARGS= 2023-08-26T13:33:09.549 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rbd --cluster cluster2 pool init mirror_parent 2023-08-26T13:33:12.550 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ '[' -n '' ']' 2023-08-26T13:33:12.550 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ CEPH_ARGS= 2023-08-26T13:33:12.551 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rbd --cluster cluster2 mirror pool enable --site-name cluster2 mirror image 2023-08-26T13:33:12.598 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rbd --cluster cluster2 mirror pool enable mirror_parent image 2023-08-26T13:33:12.638 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rbd --cluster cluster2 namespace create mirror/ns1 2023-08-26T13:33:12.673 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rbd --cluster cluster2 namespace create mirror/ns2 2023-08-26T13:33:12.709 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rbd --cluster cluster2 mirror pool enable mirror/ns1 image 2023-08-26T13:33:12.747 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rbd --cluster cluster2 mirror pool enable mirror/ns2 image 2023-08-26T13:33:12.786 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ '[' -z ']' 2023-08-26T13:33:12.786 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ '[' -z ']' 2023-08-26T13:33:12.786 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ peer_add cluster2 mirror cluster1 2023-08-26T13:33:12.787 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster2 2023-08-26T13:33:12.787 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ shift 2023-08-26T13:33:12.787 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T13:33:12.787 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ shift 2023-08-26T13:33:12.787 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local client_cluster=cluster1 2023-08-26T13:33:12.788 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ shift 2023-08-26T13:33:12.788 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local remote_cluster=cluster1 2023-08-26T13:33:12.788 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local uuid_var_name 2023-08-26T13:33:12.788 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ '[' -n '' ']' 2023-08-26T13:33:12.788 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local error_code 2023-08-26T13:33:12.789 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local peer_uuid 2023-08-26T13:33:12.789 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ for s in 1 2 4 8 16 32 2023-08-26T13:33:12.789 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ set +e 2023-08-26T13:33:12.789 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ rbd --cluster cluster2 mirror pool peer add mirror cluster1 2023-08-26T13:33:12.823 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ peer_uuid=c9991864-a605-4806-acc7-d45cc348788a 2023-08-26T13:33:12.823 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ error_code=0 2023-08-26T13:33:12.823 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ set -e 2023-08-26T13:33:12.823 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ '[' 0 -eq 17 ']' 2023-08-26T13:33:12.823 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ test 0 -eq 0 2023-08-26T13:33:12.824 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ '[' -n '' ']' 2023-08-26T13:33:12.824 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ return 0 2023-08-26T13:33:12.824 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ peer_add cluster2 mirror_parent cluster1 2023-08-26T13:33:12.824 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster2 2023-08-26T13:33:12.825 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ shift 2023-08-26T13:33:12.825 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror_parent 2023-08-26T13:33:12.825 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ shift 2023-08-26T13:33:12.825 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local client_cluster=cluster1 2023-08-26T13:33:12.825 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ shift 2023-08-26T13:33:12.826 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local remote_cluster=cluster1 2023-08-26T13:33:12.826 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local uuid_var_name 2023-08-26T13:33:12.826 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ '[' -n '' ']' 2023-08-26T13:33:12.826 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local error_code 2023-08-26T13:33:12.826 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local peer_uuid 2023-08-26T13:33:12.827 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ for s in 1 2 4 8 16 32 2023-08-26T13:33:12.827 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ set +e 2023-08-26T13:33:12.827 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ rbd --cluster cluster2 mirror pool peer add mirror_parent cluster1 2023-08-26T13:33:12.858 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ peer_uuid=890ca290-023f-4fcd-8800-d40a11e0c8e1 2023-08-26T13:33:12.858 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ error_code=0 2023-08-26T13:33:12.859 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ set -e 2023-08-26T13:33:12.859 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ '[' 0 -eq 17 ']' 2023-08-26T13:33:12.859 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ test 0 -eq 0 2023-08-26T13:33:12.859 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ '[' -n '' ']' 2023-08-26T13:33:12.859 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ return 0 2023-08-26T13:33:12.860 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ '[' -n '' ']' 2023-08-26T13:33:12.860 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ start_mirrors cluster1 2023-08-26T13:33:12.860 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster1 2023-08-26T13:33:12.860 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ seq 0 3 2023-08-26T13:33:12.879 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ for instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-08-26T13:33:12.880 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ start_mirror cluster1:0 2023-08-26T13:33:12.880 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster1:0 2023-08-26T13:33:12.880 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local instance 2023-08-26T13:33:12.880 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ set_cluster_instance cluster1:0 cluster instance 2023-08-26T13:33:12.880 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local val=cluster1:0 2023-08-26T13:33:12.880 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster_var_name=cluster 2023-08-26T13:33:12.881 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local instance_var_name=instance 2023-08-26T13:33:12.881 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ cluster=cluster1 2023-08-26T13:33:12.881 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ instance=0 2023-08-26T13:33:12.881 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ '[' 0 = cluster1:0 ']' 2023-08-26T13:33:12.881 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ eval cluster=cluster1 2023-08-26T13:33:12.882 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ cluster=cluster1 2023-08-26T13:33:12.882 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ eval instance=0 2023-08-26T13:33:12.882 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ instance=0 2023-08-26T13:33:12.882 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ test -n 1 2023-08-26T13:33:12.882 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ return 2023-08-26T13:33:12.883 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ for instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-08-26T13:33:12.883 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ start_mirror cluster1:1 2023-08-26T13:33:12.883 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster1:1 2023-08-26T13:33:12.883 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local instance 2023-08-26T13:33:12.883 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ set_cluster_instance cluster1:1 cluster instance 2023-08-26T13:33:12.884 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local val=cluster1:1 2023-08-26T13:33:12.884 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster_var_name=cluster 2023-08-26T13:33:12.884 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local instance_var_name=instance 2023-08-26T13:33:12.884 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ cluster=cluster1 2023-08-26T13:33:12.884 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ instance=1 2023-08-26T13:33:12.884 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ '[' 1 = cluster1:1 ']' 2023-08-26T13:33:12.885 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ eval cluster=cluster1 2023-08-26T13:33:12.885 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ cluster=cluster1 2023-08-26T13:33:12.885 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ eval instance=1 2023-08-26T13:33:12.885 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ instance=1 2023-08-26T13:33:12.885 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ test -n 1 2023-08-26T13:33:12.886 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ return 2023-08-26T13:33:12.886 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ for instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-08-26T13:33:12.886 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ start_mirror cluster1:2 2023-08-26T13:33:12.886 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster1:2 2023-08-26T13:33:12.886 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local instance 2023-08-26T13:33:12.887 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ set_cluster_instance cluster1:2 cluster instance 2023-08-26T13:33:12.887 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local val=cluster1:2 2023-08-26T13:33:12.887 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster_var_name=cluster 2023-08-26T13:33:12.887 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local instance_var_name=instance 2023-08-26T13:33:12.887 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ cluster=cluster1 2023-08-26T13:33:12.887 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ instance=2 2023-08-26T13:33:12.888 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ '[' 2 = cluster1:2 ']' 2023-08-26T13:33:12.888 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ eval cluster=cluster1 2023-08-26T13:33:12.888 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ cluster=cluster1 2023-08-26T13:33:12.888 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ eval instance=2 2023-08-26T13:33:12.888 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ instance=2 2023-08-26T13:33:12.889 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ test -n 1 2023-08-26T13:33:12.889 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ return 2023-08-26T13:33:12.889 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ for instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-08-26T13:33:12.889 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ start_mirror cluster1:3 2023-08-26T13:33:12.889 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster1:3 2023-08-26T13:33:12.889 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local instance 2023-08-26T13:33:12.890 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ set_cluster_instance cluster1:3 cluster instance 2023-08-26T13:33:12.890 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local val=cluster1:3 2023-08-26T13:33:12.890 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster_var_name=cluster 2023-08-26T13:33:12.890 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local instance_var_name=instance 2023-08-26T13:33:12.890 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ cluster=cluster1 2023-08-26T13:33:12.890 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ instance=3 2023-08-26T13:33:12.891 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ '[' 3 = cluster1:3 ']' 2023-08-26T13:33:12.891 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ eval cluster=cluster1 2023-08-26T13:33:12.891 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ cluster=cluster1 2023-08-26T13:33:12.891 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ eval instance=3 2023-08-26T13:33:12.891 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ instance=3 2023-08-26T13:33:12.892 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ test -n 1 2023-08-26T13:33:12.892 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ return 2023-08-26T13:33:12.892 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ start_mirrors cluster2 2023-08-26T13:33:12.892 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster2 2023-08-26T13:33:12.892 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ seq 0 3 2023-08-26T13:33:12.892 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ for instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-08-26T13:33:12.893 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ start_mirror cluster2:0 2023-08-26T13:33:12.893 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster2:0 2023-08-26T13:33:12.893 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local instance 2023-08-26T13:33:12.893 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ set_cluster_instance cluster2:0 cluster instance 2023-08-26T13:33:12.893 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local val=cluster2:0 2023-08-26T13:33:12.894 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster_var_name=cluster 2023-08-26T13:33:12.894 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local instance_var_name=instance 2023-08-26T13:33:12.894 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ cluster=cluster2 2023-08-26T13:33:12.894 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ instance=0 2023-08-26T13:33:12.894 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ '[' 0 = cluster2:0 ']' 2023-08-26T13:33:12.894 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ eval cluster=cluster2 2023-08-26T13:33:12.895 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ cluster=cluster2 2023-08-26T13:33:12.895 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ eval instance=0 2023-08-26T13:33:12.895 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ instance=0 2023-08-26T13:33:12.895 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ test -n 1 2023-08-26T13:33:12.895 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ return 2023-08-26T13:33:12.896 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ for instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-08-26T13:33:12.896 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ start_mirror cluster2:1 2023-08-26T13:33:12.896 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster2:1 2023-08-26T13:33:12.896 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local instance 2023-08-26T13:33:12.896 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ set_cluster_instance cluster2:1 cluster instance 2023-08-26T13:33:12.896 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local val=cluster2:1 2023-08-26T13:33:12.897 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster_var_name=cluster 2023-08-26T13:33:12.897 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local instance_var_name=instance 2023-08-26T13:33:12.897 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ cluster=cluster2 2023-08-26T13:33:12.897 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ instance=1 2023-08-26T13:33:12.897 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ '[' 1 = cluster2:1 ']' 2023-08-26T13:33:12.898 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ eval cluster=cluster2 2023-08-26T13:33:12.898 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ cluster=cluster2 2023-08-26T13:33:12.898 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ eval instance=1 2023-08-26T13:33:12.898 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ instance=1 2023-08-26T13:33:12.898 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ test -n 1 2023-08-26T13:33:12.899 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ return 2023-08-26T13:33:12.899 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ for instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-08-26T13:33:12.899 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ start_mirror cluster2:2 2023-08-26T13:33:12.899 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster2:2 2023-08-26T13:33:12.899 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local instance 2023-08-26T13:33:12.899 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ set_cluster_instance cluster2:2 cluster instance 2023-08-26T13:33:12.900 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local val=cluster2:2 2023-08-26T13:33:12.900 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster_var_name=cluster 2023-08-26T13:33:12.900 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local instance_var_name=instance 2023-08-26T13:33:12.900 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ cluster=cluster2 2023-08-26T13:33:12.900 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ instance=2 2023-08-26T13:33:12.900 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ '[' 2 = cluster2:2 ']' 2023-08-26T13:33:12.901 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ eval cluster=cluster2 2023-08-26T13:33:12.901 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ cluster=cluster2 2023-08-26T13:33:12.901 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ eval instance=2 2023-08-26T13:33:12.901 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ instance=2 2023-08-26T13:33:12.901 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ test -n 1 2023-08-26T13:33:12.902 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ return 2023-08-26T13:33:12.902 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ for instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-08-26T13:33:12.902 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ start_mirror cluster2:3 2023-08-26T13:33:12.902 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster2:3 2023-08-26T13:33:12.903 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local instance 2023-08-26T13:33:12.903 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ set_cluster_instance cluster2:3 cluster instance 2023-08-26T13:33:12.903 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local val=cluster2:3 2023-08-26T13:33:12.903 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster_var_name=cluster 2023-08-26T13:33:12.903 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local instance_var_name=instance 2023-08-26T13:33:12.904 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ cluster=cluster2 2023-08-26T13:33:12.904 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ instance=3 2023-08-26T13:33:12.904 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ '[' 3 = cluster2:3 ']' 2023-08-26T13:33:12.904 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ eval cluster=cluster2 2023-08-26T13:33:12.904 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ cluster=cluster2 2023-08-26T13:33:12.905 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ eval instance=3 2023-08-26T13:33:12.905 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ instance=3 2023-08-26T13:33:12.905 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ test -n 1 2023-08-26T13:33:12.905 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ return 2023-08-26T13:33:12.905 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ testlog 'TEST: add image and test replay after client crashes' 2023-08-26T13:33:12.905 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ tee -a /tmp/tmp.TfDn6BcQ3N/rbd-mirror.test.log 2023-08-26T13:33:12.906 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ date '+%F %T' 2023-08-26T13:33:12.906 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ echo 2023-08-26 13:33:12 TEST: add image and test replay after client crashes 2023-08-26T13:33:12.906 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:2023-08-26 13:33:12 TEST: add image and test replay after client crashes 2023-08-26T13:33:12.906 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ image=test 2023-08-26T13:33:12.906 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ create_image_and_enable_mirror cluster2 mirror test snapshot 512M 2023-08-26T13:33:12.907 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster2 2023-08-26T13:33:12.907 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ shift 2023-08-26T13:33:12.907 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T13:33:12.907 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ shift 2023-08-26T13:33:12.907 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=test 2023-08-26T13:33:12.908 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ shift 2023-08-26T13:33:12.908 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local mode=snapshot 2023-08-26T13:33:12.908 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ '[' -n snapshot ']' 2023-08-26T13:33:12.908 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ shift 2023-08-26T13:33:12.908 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ create_image cluster2 mirror test 512M 2023-08-26T13:33:12.909 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster2 2023-08-26T13:33:12.909 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ shift 2023-08-26T13:33:12.909 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T13:33:12.909 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ shift 2023-08-26T13:33:12.909 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=test 2023-08-26T13:33:12.910 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ shift 2023-08-26T13:33:12.910 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local size=128 2023-08-26T13:33:12.910 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ '[' -n 512M ']' 2023-08-26T13:33:12.910 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ size=512M 2023-08-26T13:33:12.910 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ shift 2023-08-26T13:33:12.911 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rbd --cluster cluster2 create --size 512M --image-feature layering,exclusive-lock mirror/test 2023-08-26T13:33:12.918 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ '[' image = image ']' 2023-08-26T13:33:12.918 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ enable_mirror cluster2 mirror test snapshot 2023-08-26T13:33:12.918 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster2 2023-08-26T13:33:12.918 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T13:33:12.918 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=test 2023-08-26T13:33:12.919 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local mode=snapshot 2023-08-26T13:33:12.919 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/test snapshot 2023-08-26T13:33:13.613 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:Mirroring enabled 2023-08-26T13:33:13.618 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rbd --cluster=cluster2 info mirror/test 2023-08-26T13:33:13.654 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:rbd image 'test': 2023-08-26T13:33:13.654 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: size 512 MiB in 128 objects 2023-08-26T13:33:13.655 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: order 22 (4 MiB objects) 2023-08-26T13:33:13.655 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: snapshot_count: 1 2023-08-26T13:33:13.655 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: id: 10aa60926f58 2023-08-26T13:33:13.655 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: block_name_prefix: rbd_data.10aa60926f58 2023-08-26T13:33:13.655 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: format: 2 2023-08-26T13:33:13.656 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: features: layering, exclusive-lock 2023-08-26T13:33:13.656 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: op_features: 2023-08-26T13:33:13.656 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: flags: 2023-08-26T13:33:13.656 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: create_timestamp: Sat Aug 26 13:33:12 2023 2023-08-26T13:33:13.656 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: access_timestamp: Sat Aug 26 13:33:12 2023 2023-08-26T13:33:13.657 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: modify_timestamp: Sat Aug 26 13:33:12 2023 2023-08-26T13:33:13.657 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: mirroring state: enabled 2023-08-26T13:33:13.657 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: mirroring mode: snapshot 2023-08-26T13:33:13.657 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: mirroring global id: e2f2554d-659e-4467-b7f8-dddd7ccb0ee9 2023-08-26T13:33:13.657 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: mirroring primary: true 2023-08-26T13:33:13.659 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ wait_for_image_replay_started cluster1 mirror test 2023-08-26T13:33:13.659 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster1 2023-08-26T13:33:13.659 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T13:33:13.659 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=test 2023-08-26T13:33:13.659 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ wait_for_image_replay_state cluster1 mirror test started 2023-08-26T13:33:13.660 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster1 2023-08-26T13:33:13.660 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T13:33:13.660 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=test 2023-08-26T13:33:13.660 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local state=started 2023-08-26T13:33:13.660 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local s 2023-08-26T13:33:13.661 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-08-26T13:33:13.661 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ sleep 1 2023-08-26T13:33:14.659 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ test_image_replay_state cluster1 mirror test started 2023-08-26T13:33:14.659 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster1 2023-08-26T13:33:14.659 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T13:33:14.660 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=test 2023-08-26T13:33:14.660 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local test_state=started 2023-08-26T13:33:14.660 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local status_result 2023-08-26T13:33:14.660 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local current_state=stopped 2023-08-26T13:33:14.660 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ admin_daemons cluster1 rbd mirror status mirror/test 2023-08-26T13:33:14.661 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local cluster_instance=cluster1 2023-08-26T13:33:14.661 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ shift 2023-08-26T13:33:14.661 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ grep -i state 2023-08-26T13:33:14.661 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local cluster=cluster1 2023-08-26T13:33:14.661 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local instance=cluster1 2023-08-26T13:33:14.661 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local loop_instance 2023-08-26T13:33:14.662 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-08-26T13:33:14.662 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ sleep 0 2023-08-26T13:33:14.662 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ '[' cluster1 '!=' cluster1 ']' 2023-08-26T13:33:14.663 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ seq 0 3 2023-08-26T13:33:14.666 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-08-26T13:33:14.666 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test 2023-08-26T13:33:14.666 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local cluster=cluster1:0 2023-08-26T13:33:14.666 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ shift 2023-08-26T13:33:14.667 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local instance 2023-08-26T13:33:14.667 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ set_cluster_instance cluster1:0 cluster instance 2023-08-26T13:33:14.667 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local val=cluster1:0 2023-08-26T13:33:14.667 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local cluster_var_name=cluster 2023-08-26T13:33:14.667 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local instance_var_name=instance 2023-08-26T13:33:14.667 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ cluster=cluster1 2023-08-26T13:33:14.668 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ instance=0 2023-08-26T13:33:14.668 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ '[' 0 = cluster1:0 ']' 2023-08-26T13:33:14.668 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ eval cluster=cluster1 2023-08-26T13:33:14.668 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ cluster=cluster1 2023-08-26T13:33:14.668 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ eval instance=0 2023-08-26T13:33:14.669 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ instance=0 2023-08-26T13:33:14.669 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ daemon_asok_file cluster1:0 cluster1 2023-08-26T13:33:14.669 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ local local_cluster=cluster1:0 2023-08-26T13:33:14.669 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ local cluster=cluster1 2023-08-26T13:33:14.669 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ local instance 2023-08-26T13:33:14.670 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2023-08-26T13:33:14.670 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ local val=cluster1:0 2023-08-26T13:33:14.670 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ local cluster_var_name=local_cluster 2023-08-26T13:33:14.670 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ local instance_var_name=instance 2023-08-26T13:33:14.670 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ cluster=cluster1 2023-08-26T13:33:14.670 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ instance=0 2023-08-26T13:33:14.671 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ '[' 0 = cluster1:0 ']' 2023-08-26T13:33:14.671 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ eval local_cluster=cluster1 2023-08-26T13:33:14.671 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++++ local_cluster=cluster1 2023-08-26T13:33:14.671 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ eval instance=0 2023-08-26T13:33:14.671 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++++ instance=0 2023-08-26T13:33:14.672 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2023-08-26T13:33:14.685 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-08-26T13:33:14.685 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-08-26T13:33:14.685 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-08-26T13:33:14.686 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2023-08-26T13:33:14.780 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:Invalid command: unused arguments: ['mirror/test'] 2023-08-26T13:33:14.781 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:rbd mirror status : get status for rbd mirror 2023-08-26T13:33:14.781 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:admin_socket: invalid command 2023-08-26T13:33:14.784 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-08-26T13:33:14.784 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/test 2023-08-26T13:33:14.784 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local cluster=cluster1:1 2023-08-26T13:33:14.784 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ shift 2023-08-26T13:33:14.785 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local instance 2023-08-26T13:33:14.785 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ set_cluster_instance cluster1:1 cluster instance 2023-08-26T13:33:14.785 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local val=cluster1:1 2023-08-26T13:33:14.785 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local cluster_var_name=cluster 2023-08-26T13:33:14.785 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local instance_var_name=instance 2023-08-26T13:33:14.786 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ cluster=cluster1 2023-08-26T13:33:14.786 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ instance=1 2023-08-26T13:33:14.786 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ '[' 1 = cluster1:1 ']' 2023-08-26T13:33:14.786 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ eval cluster=cluster1 2023-08-26T13:33:14.786 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ cluster=cluster1 2023-08-26T13:33:14.786 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ eval instance=1 2023-08-26T13:33:14.787 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ instance=1 2023-08-26T13:33:14.787 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ daemon_asok_file cluster1:1 cluster1 2023-08-26T13:33:14.787 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ local local_cluster=cluster1:1 2023-08-26T13:33:14.787 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ local cluster=cluster1 2023-08-26T13:33:14.787 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ local instance 2023-08-26T13:33:14.788 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2023-08-26T13:33:14.788 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ local val=cluster1:1 2023-08-26T13:33:14.788 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ local cluster_var_name=local_cluster 2023-08-26T13:33:14.788 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ local instance_var_name=instance 2023-08-26T13:33:14.788 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ cluster=cluster1 2023-08-26T13:33:14.788 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ instance=1 2023-08-26T13:33:14.789 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ '[' 1 = cluster1:1 ']' 2023-08-26T13:33:14.789 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ eval local_cluster=cluster1 2023-08-26T13:33:14.789 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++++ local_cluster=cluster1 2023-08-26T13:33:14.789 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ eval instance=1 2023-08-26T13:33:14.789 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++++ instance=1 2023-08-26T13:33:14.790 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2023-08-26T13:33:14.802 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-08-26T13:33:14.802 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-08-26T13:33:14.803 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-08-26T13:33:14.803 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/test 2023-08-26T13:33:14.893 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:Invalid command: unused arguments: ['mirror/test'] 2023-08-26T13:33:14.894 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:rbd mirror status : get status for rbd mirror 2023-08-26T13:33:14.894 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:admin_socket: invalid command 2023-08-26T13:33:14.897 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-08-26T13:33:14.897 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/test 2023-08-26T13:33:14.897 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local cluster=cluster1:2 2023-08-26T13:33:14.897 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ shift 2023-08-26T13:33:14.897 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local instance 2023-08-26T13:33:14.898 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ set_cluster_instance cluster1:2 cluster instance 2023-08-26T13:33:14.898 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local val=cluster1:2 2023-08-26T13:33:14.898 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local cluster_var_name=cluster 2023-08-26T13:33:14.898 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local instance_var_name=instance 2023-08-26T13:33:14.898 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ cluster=cluster1 2023-08-26T13:33:14.899 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ instance=2 2023-08-26T13:33:14.899 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ '[' 2 = cluster1:2 ']' 2023-08-26T13:33:14.899 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ eval cluster=cluster1 2023-08-26T13:33:14.899 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ cluster=cluster1 2023-08-26T13:33:14.899 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ eval instance=2 2023-08-26T13:33:14.899 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ instance=2 2023-08-26T13:33:14.900 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ daemon_asok_file cluster1:2 cluster1 2023-08-26T13:33:14.900 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ local local_cluster=cluster1:2 2023-08-26T13:33:14.900 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ local cluster=cluster1 2023-08-26T13:33:14.900 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ local instance 2023-08-26T13:33:14.900 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2023-08-26T13:33:14.901 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ local val=cluster1:2 2023-08-26T13:33:14.901 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ local cluster_var_name=local_cluster 2023-08-26T13:33:14.901 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ local instance_var_name=instance 2023-08-26T13:33:14.901 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ cluster=cluster1 2023-08-26T13:33:14.901 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ instance=2 2023-08-26T13:33:14.901 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ '[' 2 = cluster1:2 ']' 2023-08-26T13:33:14.902 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ eval local_cluster=cluster1 2023-08-26T13:33:14.902 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++++ local_cluster=cluster1 2023-08-26T13:33:14.902 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ eval instance=2 2023-08-26T13:33:14.902 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++++ instance=2 2023-08-26T13:33:14.902 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2023-08-26T13:33:14.915 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-08-26T13:33:14.915 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-08-26T13:33:14.915 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-08-26T13:33:14.915 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/test 2023-08-26T13:33:15.006 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:Invalid command: unused arguments: ['mirror/test'] 2023-08-26T13:33:15.006 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:rbd mirror status : get status for rbd mirror 2023-08-26T13:33:15.006 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:admin_socket: invalid command 2023-08-26T13:33:15.009 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-08-26T13:33:15.009 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ admin_daemon cluster1:3 rbd mirror status mirror/test 2023-08-26T13:33:15.009 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local cluster=cluster1:3 2023-08-26T13:33:15.010 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ shift 2023-08-26T13:33:15.010 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local instance 2023-08-26T13:33:15.010 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ set_cluster_instance cluster1:3 cluster instance 2023-08-26T13:33:15.010 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local val=cluster1:3 2023-08-26T13:33:15.010 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local cluster_var_name=cluster 2023-08-26T13:33:15.011 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local instance_var_name=instance 2023-08-26T13:33:15.011 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ cluster=cluster1 2023-08-26T13:33:15.011 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ instance=3 2023-08-26T13:33:15.011 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ '[' 3 = cluster1:3 ']' 2023-08-26T13:33:15.011 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ eval cluster=cluster1 2023-08-26T13:33:15.012 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ cluster=cluster1 2023-08-26T13:33:15.012 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ eval instance=3 2023-08-26T13:33:15.012 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ instance=3 2023-08-26T13:33:15.012 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ daemon_asok_file cluster1:3 cluster1 2023-08-26T13:33:15.012 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ local local_cluster=cluster1:3 2023-08-26T13:33:15.013 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ local cluster=cluster1 2023-08-26T13:33:15.013 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ local instance 2023-08-26T13:33:15.013 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ set_cluster_instance cluster1:3 local_cluster instance 2023-08-26T13:33:15.013 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ local val=cluster1:3 2023-08-26T13:33:15.013 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ local cluster_var_name=local_cluster 2023-08-26T13:33:15.013 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ local instance_var_name=instance 2023-08-26T13:33:15.014 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ cluster=cluster1 2023-08-26T13:33:15.014 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ instance=3 2023-08-26T13:33:15.014 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ '[' 3 = cluster1:3 ']' 2023-08-26T13:33:15.014 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ eval local_cluster=cluster1 2023-08-26T13:33:15.014 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++++ local_cluster=cluster1 2023-08-26T13:33:15.015 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ eval instance=3 2023-08-26T13:33:15.015 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++++ instance=3 2023-08-26T13:33:15.015 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2023-08-26T13:33:15.027 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2023-08-26T13:33:15.027 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2023-08-26T13:33:15.027 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2023-08-26T13:33:15.027 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror status mirror/test 2023-08-26T13:33:15.118 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:Invalid command: unused arguments: ['mirror/test'] 2023-08-26T13:33:15.118 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:rbd mirror status : get status for rbd mirror 2023-08-26T13:33:15.119 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:admin_socket: invalid command 2023-08-26T13:33:15.119 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-08-26T13:33:15.119 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ sleep 1 2023-08-26T13:33:16.120 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ '[' cluster1 '!=' cluster1 ']' 2023-08-26T13:33:16.121 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ seq 0 3 2023-08-26T13:33:16.123 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-08-26T13:33:16.123 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test 2023-08-26T13:33:16.124 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local cluster=cluster1:0 2023-08-26T13:33:16.124 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ shift 2023-08-26T13:33:16.124 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local instance 2023-08-26T13:33:16.124 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ set_cluster_instance cluster1:0 cluster instance 2023-08-26T13:33:16.124 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local val=cluster1:0 2023-08-26T13:33:16.125 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local cluster_var_name=cluster 2023-08-26T13:33:16.125 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local instance_var_name=instance 2023-08-26T13:33:16.125 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ cluster=cluster1 2023-08-26T13:33:16.125 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ instance=0 2023-08-26T13:33:16.125 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ '[' 0 = cluster1:0 ']' 2023-08-26T13:33:16.126 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ eval cluster=cluster1 2023-08-26T13:33:16.126 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ cluster=cluster1 2023-08-26T13:33:16.126 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ eval instance=0 2023-08-26T13:33:16.126 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ instance=0 2023-08-26T13:33:16.126 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ daemon_asok_file cluster1:0 cluster1 2023-08-26T13:33:16.127 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ local local_cluster=cluster1:0 2023-08-26T13:33:16.127 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ local cluster=cluster1 2023-08-26T13:33:16.127 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ local instance 2023-08-26T13:33:16.127 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2023-08-26T13:33:16.127 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ local val=cluster1:0 2023-08-26T13:33:16.128 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ local cluster_var_name=local_cluster 2023-08-26T13:33:16.128 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ local instance_var_name=instance 2023-08-26T13:33:16.128 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ cluster=cluster1 2023-08-26T13:33:16.128 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ instance=0 2023-08-26T13:33:16.128 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ '[' 0 = cluster1:0 ']' 2023-08-26T13:33:16.129 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ eval local_cluster=cluster1 2023-08-26T13:33:16.129 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++++ local_cluster=cluster1 2023-08-26T13:33:16.129 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ eval instance=0 2023-08-26T13:33:16.129 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++++ instance=0 2023-08-26T13:33:16.129 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2023-08-26T13:33:16.142 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-08-26T13:33:16.142 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-08-26T13:33:16.143 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-08-26T13:33:16.143 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2023-08-26T13:33:16.236 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:Invalid command: unused arguments: ['mirror/test'] 2023-08-26T13:33:16.237 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:rbd mirror status : get status for rbd mirror 2023-08-26T13:33:16.237 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:admin_socket: invalid command 2023-08-26T13:33:16.239 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-08-26T13:33:16.240 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/test 2023-08-26T13:33:16.240 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local cluster=cluster1:1 2023-08-26T13:33:16.240 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ shift 2023-08-26T13:33:16.240 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local instance 2023-08-26T13:33:16.240 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ set_cluster_instance cluster1:1 cluster instance 2023-08-26T13:33:16.240 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local val=cluster1:1 2023-08-26T13:33:16.241 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local cluster_var_name=cluster 2023-08-26T13:33:16.241 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local instance_var_name=instance 2023-08-26T13:33:16.241 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ cluster=cluster1 2023-08-26T13:33:16.241 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ instance=1 2023-08-26T13:33:16.241 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ '[' 1 = cluster1:1 ']' 2023-08-26T13:33:16.242 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ eval cluster=cluster1 2023-08-26T13:33:16.242 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ cluster=cluster1 2023-08-26T13:33:16.242 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ eval instance=1 2023-08-26T13:33:16.242 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ instance=1 2023-08-26T13:33:16.242 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ daemon_asok_file cluster1:1 cluster1 2023-08-26T13:33:16.243 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ local local_cluster=cluster1:1 2023-08-26T13:33:16.243 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ local cluster=cluster1 2023-08-26T13:33:16.243 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ local instance 2023-08-26T13:33:16.243 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2023-08-26T13:33:16.243 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ local val=cluster1:1 2023-08-26T13:33:16.243 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ local cluster_var_name=local_cluster 2023-08-26T13:33:16.244 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ local instance_var_name=instance 2023-08-26T13:33:16.244 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ cluster=cluster1 2023-08-26T13:33:16.244 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ instance=1 2023-08-26T13:33:16.244 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ '[' 1 = cluster1:1 ']' 2023-08-26T13:33:16.244 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ eval local_cluster=cluster1 2023-08-26T13:33:16.245 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++++ local_cluster=cluster1 2023-08-26T13:33:16.245 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ eval instance=1 2023-08-26T13:33:16.245 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++++ instance=1 2023-08-26T13:33:16.245 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2023-08-26T13:33:16.258 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-08-26T13:33:16.258 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-08-26T13:33:16.258 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-08-26T13:33:16.259 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/test 2023-08-26T13:33:16.352 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:Invalid command: unused arguments: ['mirror/test'] 2023-08-26T13:33:16.352 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:rbd mirror status : get status for rbd mirror 2023-08-26T13:33:16.352 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:admin_socket: invalid command 2023-08-26T13:33:16.355 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-08-26T13:33:16.355 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/test 2023-08-26T13:33:16.355 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local cluster=cluster1:2 2023-08-26T13:33:16.356 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ shift 2023-08-26T13:33:16.356 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local instance 2023-08-26T13:33:16.356 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ set_cluster_instance cluster1:2 cluster instance 2023-08-26T13:33:16.356 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local val=cluster1:2 2023-08-26T13:33:16.356 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local cluster_var_name=cluster 2023-08-26T13:33:16.357 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local instance_var_name=instance 2023-08-26T13:33:16.357 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ cluster=cluster1 2023-08-26T13:33:16.357 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ instance=2 2023-08-26T13:33:16.357 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ '[' 2 = cluster1:2 ']' 2023-08-26T13:33:16.357 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ eval cluster=cluster1 2023-08-26T13:33:16.358 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ cluster=cluster1 2023-08-26T13:33:16.358 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ eval instance=2 2023-08-26T13:33:16.358 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ instance=2 2023-08-26T13:33:16.358 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ daemon_asok_file cluster1:2 cluster1 2023-08-26T13:33:16.358 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ local local_cluster=cluster1:2 2023-08-26T13:33:16.359 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ local cluster=cluster1 2023-08-26T13:33:16.359 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ local instance 2023-08-26T13:33:16.359 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2023-08-26T13:33:16.359 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ local val=cluster1:2 2023-08-26T13:33:16.359 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ local cluster_var_name=local_cluster 2023-08-26T13:33:16.360 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ local instance_var_name=instance 2023-08-26T13:33:16.360 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ cluster=cluster1 2023-08-26T13:33:16.360 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ instance=2 2023-08-26T13:33:16.360 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ '[' 2 = cluster1:2 ']' 2023-08-26T13:33:16.360 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ eval local_cluster=cluster1 2023-08-26T13:33:16.361 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++++ local_cluster=cluster1 2023-08-26T13:33:16.361 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ eval instance=2 2023-08-26T13:33:16.361 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++++ instance=2 2023-08-26T13:33:16.361 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2023-08-26T13:33:16.373 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-08-26T13:33:16.374 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-08-26T13:33:16.374 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-08-26T13:33:16.374 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/test 2023-08-26T13:33:16.468 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:Invalid command: unused arguments: ['mirror/test'] 2023-08-26T13:33:16.469 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:rbd mirror status : get status for rbd mirror 2023-08-26T13:33:16.469 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:admin_socket: invalid command 2023-08-26T13:33:16.472 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-08-26T13:33:16.472 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ admin_daemon cluster1:3 rbd mirror status mirror/test 2023-08-26T13:33:16.472 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local cluster=cluster1:3 2023-08-26T13:33:16.472 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ shift 2023-08-26T13:33:16.472 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local instance 2023-08-26T13:33:16.473 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ set_cluster_instance cluster1:3 cluster instance 2023-08-26T13:33:16.473 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local val=cluster1:3 2023-08-26T13:33:16.473 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local cluster_var_name=cluster 2023-08-26T13:33:16.473 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local instance_var_name=instance 2023-08-26T13:33:16.473 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ cluster=cluster1 2023-08-26T13:33:16.474 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ instance=3 2023-08-26T13:33:16.474 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ '[' 3 = cluster1:3 ']' 2023-08-26T13:33:16.474 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ eval cluster=cluster1 2023-08-26T13:33:16.474 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ cluster=cluster1 2023-08-26T13:33:16.474 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ eval instance=3 2023-08-26T13:33:16.474 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ instance=3 2023-08-26T13:33:16.475 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ daemon_asok_file cluster1:3 cluster1 2023-08-26T13:33:16.475 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ local local_cluster=cluster1:3 2023-08-26T13:33:16.475 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ local cluster=cluster1 2023-08-26T13:33:16.475 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ local instance 2023-08-26T13:33:16.475 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ set_cluster_instance cluster1:3 local_cluster instance 2023-08-26T13:33:16.476 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ local val=cluster1:3 2023-08-26T13:33:16.476 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ local cluster_var_name=local_cluster 2023-08-26T13:33:16.476 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ local instance_var_name=instance 2023-08-26T13:33:16.476 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ cluster=cluster1 2023-08-26T13:33:16.476 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ instance=3 2023-08-26T13:33:16.477 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ '[' 3 = cluster1:3 ']' 2023-08-26T13:33:16.477 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ eval local_cluster=cluster1 2023-08-26T13:33:16.477 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++++ local_cluster=cluster1 2023-08-26T13:33:16.477 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ eval instance=3 2023-08-26T13:33:16.477 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++++ instance=3 2023-08-26T13:33:16.478 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2023-08-26T13:33:16.489 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2023-08-26T13:33:16.490 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2023-08-26T13:33:16.490 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2023-08-26T13:33:16.490 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror status mirror/test 2023-08-26T13:33:16.584 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:Invalid command: unused arguments: ['mirror/test'] 2023-08-26T13:33:16.584 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:rbd mirror status : get status for rbd mirror 2023-08-26T13:33:16.584 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:admin_socket: invalid command 2023-08-26T13:33:16.585 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-08-26T13:33:16.585 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ sleep 2 2023-08-26T13:33:18.586 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ '[' cluster1 '!=' cluster1 ']' 2023-08-26T13:33:18.586 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ seq 0 3 2023-08-26T13:33:18.589 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-08-26T13:33:18.590 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test 2023-08-26T13:33:18.590 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local cluster=cluster1:0 2023-08-26T13:33:18.590 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ shift 2023-08-26T13:33:18.590 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local instance 2023-08-26T13:33:18.590 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ set_cluster_instance cluster1:0 cluster instance 2023-08-26T13:33:18.591 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local val=cluster1:0 2023-08-26T13:33:18.591 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local cluster_var_name=cluster 2023-08-26T13:33:18.591 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local instance_var_name=instance 2023-08-26T13:33:18.591 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ cluster=cluster1 2023-08-26T13:33:18.591 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ instance=0 2023-08-26T13:33:18.592 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ '[' 0 = cluster1:0 ']' 2023-08-26T13:33:18.592 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ eval cluster=cluster1 2023-08-26T13:33:18.592 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ cluster=cluster1 2023-08-26T13:33:18.592 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ eval instance=0 2023-08-26T13:33:18.592 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ instance=0 2023-08-26T13:33:18.593 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ daemon_asok_file cluster1:0 cluster1 2023-08-26T13:33:18.593 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ local local_cluster=cluster1:0 2023-08-26T13:33:18.593 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ local cluster=cluster1 2023-08-26T13:33:18.593 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ local instance 2023-08-26T13:33:18.593 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2023-08-26T13:33:18.593 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ local val=cluster1:0 2023-08-26T13:33:18.594 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ local cluster_var_name=local_cluster 2023-08-26T13:33:18.594 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ local instance_var_name=instance 2023-08-26T13:33:18.594 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ cluster=cluster1 2023-08-26T13:33:18.594 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ instance=0 2023-08-26T13:33:18.594 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ '[' 0 = cluster1:0 ']' 2023-08-26T13:33:18.595 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ eval local_cluster=cluster1 2023-08-26T13:33:18.595 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++++ local_cluster=cluster1 2023-08-26T13:33:18.595 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ eval instance=0 2023-08-26T13:33:18.595 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++++ instance=0 2023-08-26T13:33:18.595 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2023-08-26T13:33:18.611 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-08-26T13:33:18.611 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-08-26T13:33:18.612 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-08-26T13:33:18.612 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2023-08-26T13:33:18.722 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ return 0 2023-08-26T13:33:18.722 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ status_result=' "state": "Replaying"' 2023-08-26T13:33:18.723 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ echo ' "state": "Replaying"' 2023-08-26T13:33:18.723 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ grep -i Replaying 2023-08-26T13:33:18.724 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: "state": "Replaying" 2023-08-26T13:33:18.725 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ current_state=started 2023-08-26T13:33:18.725 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ test started = started 2023-08-26T13:33:18.725 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ return 0 2023-08-26T13:33:18.725 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ clean_snap_name= 2023-08-26T13:33:18.726 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ seq 1 10 2023-08-26T13:33:18.726 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ for i in `seq 1 10` 2023-08-26T13:33:18.726 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ stress_write_image cluster2 mirror test 2023-08-26T13:33:18.726 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster2 2023-08-26T13:33:18.726 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T13:33:18.727 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=test 2023-08-26T13:33:18.727 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2023-08-26T13:33:18.727 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local duration=13 2023-08-26T13:33:18.728 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ set +e 2023-08-26T13:33:18.728 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ timeout 13s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror test --debug-rbd=20 --debug-journaler=20 2023-08-26T13:33:18.921 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:100 IOs, 49344512 bytes 2023-08-26T13:33:19.025 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:200 IOs, 66906112 bytes 2023-08-26T13:33:19.228 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:300 IOs, 150300672 bytes 2023-08-26T13:33:19.443 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:400 IOs, 219578368 bytes 2023-08-26T13:33:19.563 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:500 IOs, 248800256 bytes 2023-08-26T13:33:19.739 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:600 IOs, 307806208 bytes 2023-08-26T13:33:19.825 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:700 IOs, 319423488 bytes 2023-08-26T13:33:19.987 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:800 IOs, 375872512 bytes 2023-08-26T13:33:20.137 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:900 IOs, 415940608 bytes 2023-08-26T13:33:20.276 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:1000 IOs, 451723264 bytes 2023-08-26T13:33:20.399 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:1100 IOs, 475853824 bytes 2023-08-26T13:33:20.571 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:1200 IOs, 529098752 bytes 2023-08-26T13:33:20.678 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:1300 IOs, 549132288 bytes 2023-08-26T13:33:20.810 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:1400 IOs, 588793856 bytes 2023-08-26T13:33:21.004 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:1500 IOs, 655696896 bytes 2023-08-26T13:33:21.127 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:1600 IOs, 684451840 bytes 2023-08-26T13:33:21.333 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:1700 IOs, 755018752 bytes 2023-08-26T13:33:21.451 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:1800 IOs, 781795328 bytes 2023-08-26T13:33:21.617 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:1900 IOs, 839507968 bytes 2023-08-26T13:33:21.700 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:2000 IOs, 853869568 bytes 2023-08-26T13:33:21.856 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:2100 IOs, 906711040 bytes 2023-08-26T13:33:21.995 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:2200 IOs, 945617920 bytes 2023-08-26T13:33:22.119 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:2300 IOs, 973536256 bytes 2023-08-26T13:33:22.319 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:2400 IOs, 1038920704 bytes 2023-08-26T13:33:22.417 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:2500 IOs, 1046742016 bytes 2023-08-26T13:33:22.468 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:2600 IOs, 1047314432 bytes 2023-08-26T13:33:22.633 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:2700 IOs, 1097354240 bytes 2023-08-26T13:33:22.748 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:2800 IOs, 1118810112 bytes 2023-08-26T13:33:22.827 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:2900 IOs, 1119450112 bytes 2023-08-26T13:33:22.979 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:3000 IOs, 1161917440 bytes 2023-08-26T13:33:23.063 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:3100 IOs, 1185754112 bytes 2023-08-26T13:33:23.228 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:3200 IOs, 1238377472 bytes 2023-08-26T13:33:23.435 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:3300 IOs, 1318105088 bytes 2023-08-26T13:33:23.582 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:3400 IOs, 1347322880 bytes 2023-08-26T13:33:23.714 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:3500 IOs, 1383006208 bytes 2023-08-26T13:33:23.856 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:3600 IOs, 1422909440 bytes 2023-08-26T13:33:24.035 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:3700 IOs, 1482962944 bytes 2023-08-26T13:33:24.206 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:3800 IOs, 1534024704 bytes 2023-08-26T13:33:24.282 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:3900 IOs, 1549040640 bytes 2023-08-26T13:33:24.385 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:4000 IOs, 1566517248 bytes 2023-08-26T13:33:24.507 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:4100 IOs, 1587807232 bytes 2023-08-26T13:33:24.775 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:4200 IOs, 1687435264 bytes 2023-08-26T13:33:24.961 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:4300 IOs, 1749458944 bytes 2023-08-26T13:33:25.124 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:4400 IOs, 1798350848 bytes 2023-08-26T13:33:25.228 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:4500 IOs, 1823205376 bytes 2023-08-26T13:33:25.353 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:4600 IOs, 1868794880 bytes 2023-08-26T13:33:25.514 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:4700 IOs, 1904253952 bytes 2023-08-26T13:33:25.648 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:4800 IOs, 1959788544 bytes 2023-08-26T13:33:25.758 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:4900 IOs, 1971509248 bytes 2023-08-26T13:33:25.966 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:5000 IOs, 2051553280 bytes 2023-08-26T13:33:26.142 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:5100 IOs, 2108915712 bytes 2023-08-26T13:33:26.222 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:5200 IOs, 2109516800 bytes 2023-08-26T13:33:26.377 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:5300 IOs, 2150191104 bytes 2023-08-26T13:33:26.438 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:5400 IOs, 2150807552 bytes 2023-08-26T13:33:26.597 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:5500 IOs, 2194643968 bytes 2023-08-26T13:33:26.748 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:5600 IOs, 2244936704 bytes 2023-08-26T13:33:26.878 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:5700 IOs, 2282296320 bytes 2023-08-26T13:33:27.088 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:5800 IOs, 2342930432 bytes 2023-08-26T13:33:27.300 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:5900 IOs, 2416258048 bytes 2023-08-26T13:33:27.517 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:6000 IOs, 2496974848 bytes 2023-08-26T13:33:27.594 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:6100 IOs, 2503281664 bytes 2023-08-26T13:33:27.659 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:6200 IOs, 2503896064 bytes 2023-08-26T13:33:27.813 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:6300 IOs, 2547075072 bytes 2023-08-26T13:33:27.944 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:6400 IOs, 2584556544 bytes 2023-08-26T13:33:28.092 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:6500 IOs, 2623309824 bytes 2023-08-26T13:33:28.374 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:6600 IOs, 2737912832 bytes 2023-08-26T13:33:28.639 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:6700 IOs, 2795089920 bytes 2023-08-26T13:33:28.717 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:6800 IOs, 2803348480 bytes 2023-08-26T13:33:28.857 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:6900 IOs, 2839924736 bytes 2023-08-26T13:33:29.038 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:7000 IOs, 2900702208 bytes 2023-08-26T13:33:29.211 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:7100 IOs, 2961596416 bytes 2023-08-26T13:33:29.369 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:7200 IOs, 3003484160 bytes 2023-08-26T13:33:29.492 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:7300 IOs, 3030054912 bytes 2023-08-26T13:33:29.639 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:7400 IOs, 3071937536 bytes 2023-08-26T13:33:29.798 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:7500 IOs, 3114170368 bytes 2023-08-26T13:33:29.884 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:7600 IOs, 3126728704 bytes 2023-08-26T13:33:30.076 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:7700 IOs, 3196554240 bytes 2023-08-26T13:33:30.213 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:7800 IOs, 3241459712 bytes 2023-08-26T13:33:30.319 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:7900 IOs, 3281291264 bytes 2023-08-26T13:33:30.528 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:8000 IOs, 3315756032 bytes 2023-08-26T13:33:30.648 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:8100 IOs, 3316392960 bytes 2023-08-26T13:33:30.789 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:8200 IOs, 3370100736 bytes 2023-08-26T13:33:31.014 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:8300 IOs, 3437685760 bytes 2023-08-26T13:33:31.088 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:8400 IOs, 3440919552 bytes 2023-08-26T13:33:31.254 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:8500 IOs, 3493507072 bytes 2023-08-26T13:33:31.394 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:8600 IOs, 3538286592 bytes 2023-08-26T13:33:31.540 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:8700 IOs, 3575662592 bytes 2023-08-26T13:33:31.652 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:8800 IOs, 3601179648 bytes 2023-08-26T13:33:31.730 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ error_code=124 2023-08-26T13:33:31.731 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ set -e 2023-08-26T13:33:31.731 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ '[' 124 -eq 124 ']' 2023-08-26T13:33:31.731 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ return 0 2023-08-26T13:33:31.731 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ wait_for_status_in_pool_dir cluster1 mirror test up+replaying 2023-08-26T13:33:31.732 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster1 2023-08-26T13:33:31.732 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T13:33:31.732 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=test 2023-08-26T13:33:31.732 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local state_pattern=up+replaying 2023-08-26T13:33:31.732 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local description_pattern= 2023-08-26T13:33:31.733 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local service_pattern= 2023-08-26T13:33:31.733 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-08-26T13:33:31.733 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ sleep 1 2023-08-26T13:33:32.732 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ test_status_in_pool_dir cluster1 mirror test up+replaying '' '' 2023-08-26T13:33:32.732 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster1 2023-08-26T13:33:32.733 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T13:33:32.733 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=test 2023-08-26T13:33:32.733 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local state_pattern=up+replaying 2023-08-26T13:33:32.733 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local description_pattern= 2023-08-26T13:33:32.733 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local service_pattern= 2023-08-26T13:33:32.734 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ mkfname cluster1-mirror-test.mirror_status 2023-08-26T13:33:32.734 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ echo cluster1-mirror-test.mirror_status 2023-08-26T13:33:32.734 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ sed -e 's|[/ ]|_|g' 2023-08-26T13:33:32.735 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local status_log=/tmp/tmp.TfDn6BcQ3N/cluster1-mirror-test.mirror_status 2023-08-26T13:33:32.735 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ CEPH_ARGS= 2023-08-26T13:33:32.735 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rbd --cluster cluster1 mirror image status mirror/test 2023-08-26T13:33:32.735 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ tee /tmp/tmp.TfDn6BcQ3N/cluster1-mirror-test.mirror_status 2023-08-26T13:33:32.782 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:test: 2023-08-26T13:33:32.783 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr: global_id: e2f2554d-659e-4467-b7f8-dddd7ccb0ee9 2023-08-26T13:33:32.783 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr: state: down+unknown 2023-08-26T13:33:32.783 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr: description: status not found 2023-08-26T13:33:32.783 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr: last_update: 2023-08-26T13:33:32.783 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr: peer_sites: 2023-08-26T13:33:32.784 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr: name: cluster2 2023-08-26T13:33:32.784 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr: state: down+unknown 2023-08-26T13:33:32.784 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr: description: status not found 2023-08-26T13:33:32.784 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr: last_update: 2023-08-26T13:33:32.787 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ grep '^ state: .*up+replaying' /tmp/tmp.TfDn6BcQ3N/cluster1-mirror-test.mirror_status 2023-08-26T13:33:32.789 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ return 1 2023-08-26T13:33:32.789 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-08-26T13:33:32.789 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ sleep 2 2023-08-26T13:33:34.791 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ test_status_in_pool_dir cluster1 mirror test up+replaying '' '' 2023-08-26T13:33:34.791 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster1 2023-08-26T13:33:34.791 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T13:33:34.791 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=test 2023-08-26T13:33:34.792 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local state_pattern=up+replaying 2023-08-26T13:33:34.792 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local description_pattern= 2023-08-26T13:33:34.792 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local service_pattern= 2023-08-26T13:33:34.792 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ mkfname cluster1-mirror-test.mirror_status 2023-08-26T13:33:34.793 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ echo cluster1-mirror-test.mirror_status 2023-08-26T13:33:34.793 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ sed -e 's|[/ ]|_|g' 2023-08-26T13:33:34.793 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local status_log=/tmp/tmp.TfDn6BcQ3N/cluster1-mirror-test.mirror_status 2023-08-26T13:33:34.794 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ CEPH_ARGS= 2023-08-26T13:33:34.794 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rbd --cluster cluster1 mirror image status mirror/test 2023-08-26T13:33:34.794 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ tee /tmp/tmp.TfDn6BcQ3N/cluster1-mirror-test.mirror_status 2023-08-26T13:33:34.839 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:test: 2023-08-26T13:33:34.839 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr: global_id: e2f2554d-659e-4467-b7f8-dddd7ccb0ee9 2023-08-26T13:33:34.840 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr: state: down+unknown 2023-08-26T13:33:34.840 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr: description: status not found 2023-08-26T13:33:34.840 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr: last_update: 2023-08-26T13:33:34.840 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr: peer_sites: 2023-08-26T13:33:34.840 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr: name: cluster2 2023-08-26T13:33:34.841 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr: state: down+unknown 2023-08-26T13:33:34.841 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr: description: status not found 2023-08-26T13:33:34.841 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr: last_update: 2023-08-26T13:33:34.844 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ grep '^ state: .*up+replaying' /tmp/tmp.TfDn6BcQ3N/cluster1-mirror-test.mirror_status 2023-08-26T13:33:34.845 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ return 1 2023-08-26T13:33:34.845 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-08-26T13:33:34.845 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ sleep 4 2023-08-26T13:33:38.847 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ test_status_in_pool_dir cluster1 mirror test up+replaying '' '' 2023-08-26T13:33:38.847 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster1 2023-08-26T13:33:38.847 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T13:33:38.847 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=test 2023-08-26T13:33:38.848 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local state_pattern=up+replaying 2023-08-26T13:33:38.848 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local description_pattern= 2023-08-26T13:33:38.848 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local service_pattern= 2023-08-26T13:33:38.848 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ mkfname cluster1-mirror-test.mirror_status 2023-08-26T13:33:38.848 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ echo cluster1-mirror-test.mirror_status 2023-08-26T13:33:38.849 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ sed -e 's|[/ ]|_|g' 2023-08-26T13:33:38.849 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local status_log=/tmp/tmp.TfDn6BcQ3N/cluster1-mirror-test.mirror_status 2023-08-26T13:33:38.849 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ CEPH_ARGS= 2023-08-26T13:33:38.850 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rbd --cluster cluster1 mirror image status mirror/test 2023-08-26T13:33:38.850 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ tee /tmp/tmp.TfDn6BcQ3N/cluster1-mirror-test.mirror_status 2023-08-26T13:33:38.895 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:test: 2023-08-26T13:33:38.896 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr: global_id: e2f2554d-659e-4467-b7f8-dddd7ccb0ee9 2023-08-26T13:33:38.896 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr: state: down+unknown 2023-08-26T13:33:38.896 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr: description: status not found 2023-08-26T13:33:38.897 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr: last_update: 2023-08-26T13:33:38.897 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr: peer_sites: 2023-08-26T13:33:38.897 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr: name: cluster2 2023-08-26T13:33:38.897 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr: state: down+unknown 2023-08-26T13:33:38.897 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr: description: status not found 2023-08-26T13:33:38.898 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr: last_update: 2023-08-26T13:33:38.900 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ grep '^ state: .*up+replaying' /tmp/tmp.TfDn6BcQ3N/cluster1-mirror-test.mirror_status 2023-08-26T13:33:38.901 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ return 1 2023-08-26T13:33:38.902 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-08-26T13:33:38.902 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ sleep 8 2023-08-26T13:33:45.444 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.0 with skip (0.7038971938595929) > weight (0.25) 2023-08-26T13:33:45.444 INFO:tasks.rbd_mirror_thrash:kill cluster1.client.mirror.1 2023-08-26T13:33:45.445 INFO:tasks.rbd_mirror.cluster1.client.mirror.1:Sent signal 15 2023-08-26T13:33:45.445 INFO:tasks.rbd_mirror_thrash:waiting for 1 secs before reviving daemons 2023-08-26T13:33:45.446 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi124.stderr:2023-08-26T13:33:45.444+0000 7efc17daf700 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.1 (PID: 105972) UID: 1000 2023-08-26T13:33:46.446 INFO:tasks.rbd_mirror_thrash:waiting for cluster1.client.mirror.1 2023-08-26T13:33:46.446 INFO:teuthology.orchestra.run:waiting for 600 2023-08-26T13:33:46.447 INFO:tasks.rbd_mirror_thrash:reviving cluster1.client.mirror.1 2023-08-26T13:33:46.447 INFO:tasks.rbd_mirror.cluster1.client.mirror.1:Restarting daemon 2023-08-26T13:33:46.447 DEBUG:teuthology.orchestra.run.smithi124:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.1 2023-08-26T13:33:46.450 INFO:tasks.rbd_mirror.cluster1.client.mirror.1:Started 2023-08-26T13:33:46.451 INFO:tasks.rbd_mirror_thrash:waiting for 80 secs before thrashing 2023-08-26T13:33:46.903 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ test_status_in_pool_dir cluster1 mirror test up+replaying '' '' 2023-08-26T13:33:46.903 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster1 2023-08-26T13:33:46.904 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T13:33:46.904 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=test 2023-08-26T13:33:46.904 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local state_pattern=up+replaying 2023-08-26T13:33:46.904 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local description_pattern= 2023-08-26T13:33:46.904 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local service_pattern= 2023-08-26T13:33:46.905 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ mkfname cluster1-mirror-test.mirror_status 2023-08-26T13:33:46.905 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ echo cluster1-mirror-test.mirror_status 2023-08-26T13:33:46.906 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ sed -e 's|[/ ]|_|g' 2023-08-26T13:33:46.906 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local status_log=/tmp/tmp.TfDn6BcQ3N/cluster1-mirror-test.mirror_status 2023-08-26T13:33:46.906 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ CEPH_ARGS= 2023-08-26T13:33:46.906 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rbd --cluster cluster1 mirror image status mirror/test 2023-08-26T13:33:46.906 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ tee /tmp/tmp.TfDn6BcQ3N/cluster1-mirror-test.mirror_status 2023-08-26T13:33:46.956 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:test: 2023-08-26T13:33:46.956 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr: global_id: e2f2554d-659e-4467-b7f8-dddd7ccb0ee9 2023-08-26T13:33:46.957 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr: state: up+replaying 2023-08-26T13:33:46.957 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr: description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":0.0,"last_snapshot_bytes":0,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1693056793,"remote_snapshot_timestamp":1693056793,"replay_state":"idle"} 2023-08-26T13:33:46.957 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr: service: mirror.0 on smithi124 2023-08-26T13:33:46.957 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr: last_update: 2023-08-26 13:33:45 2023-08-26T13:33:46.958 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr: peer_sites: 2023-08-26T13:33:46.958 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr: name: cluster2 2023-08-26T13:33:46.958 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr: state: up+stopped 2023-08-26T13:33:46.958 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr: description: local image is primary 2023-08-26T13:33:46.958 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr: last_update: 2023-08-26 13:33:45 2023-08-26T13:33:46.961 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ grep '^ state: .*up+replaying' /tmp/tmp.TfDn6BcQ3N/cluster1-mirror-test.mirror_status 2023-08-26T13:33:46.962 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: state: up+replaying 2023-08-26T13:33:46.962 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ grep '^ description: .*' /tmp/tmp.TfDn6BcQ3N/cluster1-mirror-test.mirror_status 2023-08-26T13:33:46.963 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":0.0,"last_snapshot_bytes":0,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1693056793,"remote_snapshot_timestamp":1693056793,"replay_state":"idle"} 2023-08-26T13:33:46.964 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ '[' -n '' ']' 2023-08-26T13:33:46.964 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ echo up+replaying 2023-08-26T13:33:46.964 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ grep '^up+' 2023-08-26T13:33:46.965 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:up+replaying 2023-08-26T13:33:46.965 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ grep 'service: *mirror..* on ' /tmp/tmp.TfDn6BcQ3N/cluster1-mirror-test.mirror_status 2023-08-26T13:33:46.966 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: service: mirror.0 on smithi124 2023-08-26T13:33:46.966 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' /tmp/tmp.TfDn6BcQ3N/cluster1-mirror-test.mirror_status 2023-08-26T13:33:46.967 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local 'last_update=2023-08-26 13:33:45' 2023-08-26T13:33:46.967 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ test_mirror_pool_status_verbose cluster1 mirror test up+replaying '2023-08-26 13:33:45' 2023-08-26T13:33:46.968 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster1 2023-08-26T13:33:46.968 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T13:33:46.968 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=test 2023-08-26T13:33:46.968 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local state_pattern=up+replaying 2023-08-26T13:33:46.969 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local 'prev_last_update=2023-08-26 13:33:45' 2023-08-26T13:33:46.969 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ mkfname cluster1-mirror.mirror_status 2023-08-26T13:33:46.969 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ echo cluster1-mirror.mirror_status 2023-08-26T13:33:46.969 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ sed -e 's|[/ ]|_|g' 2023-08-26T13:33:46.970 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local status_log=/tmp/tmp.TfDn6BcQ3N/cluster1-mirror.mirror_status 2023-08-26T13:33:46.970 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2023-08-26T13:33:47.006 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:rbd: failed to query services: (13) Permission denied 2023-08-26T13:33:47.028 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local last_update state 2023-08-26T13:33:47.028 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update' 2023-08-26T13:33:47.048 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ last_update='2023-08-26 13:33:45' 2023-08-26T13:33:47.048 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state' 2023-08-26T13:33:47.049 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ state=up+replaying 2023-08-26T13:33:47.049 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ echo up+replaying 2023-08-26T13:33:47.050 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ grep up+replaying 2023-08-26T13:33:47.050 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:up+replaying 2023-08-26T13:33:47.051 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ return 0 2023-08-26T13:33:47.051 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ return 0 2023-08-26T13:33:47.052 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ snap_name=snap1 2023-08-26T13:33:47.052 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ create_snap cluster2 mirror test snap1 2023-08-26T13:33:47.052 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster2 2023-08-26T13:33:47.052 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T13:33:47.052 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=test 2023-08-26T13:33:47.053 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local snap_name=snap1 2023-08-26T13:33:47.053 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rbd --cluster cluster2 -p mirror snap create test@snap1 --debug-rbd=20 --debug-journaler=20 2023-08-26T13:33:56.445 INFO:tasks.rbd_mirror_thrash:kill cluster2.client.mirror.0 2023-08-26T13:33:56.445 INFO:tasks.rbd_mirror.cluster2.client.mirror.0:Sent signal 15 2023-08-26T13:33:56.446 INFO:tasks.rbd_mirror_thrash:waiting for 8 secs before reviving daemons 2023-08-26T13:33:56.446 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi124.stderr:2023-08-26T13:33:56.445+0000 7f5c4053c700 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.0 (PID: 106047) UID: 1000 2023-08-26T13:34:00.609 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ wait_for_image_replay_started cluster1 mirror test 2023-08-26T13:34:00.609 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster1 2023-08-26T13:34:00.609 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T13:34:00.610 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=test 2023-08-26T13:34:00.610 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ wait_for_image_replay_state cluster1 mirror test started 2023-08-26T13:34:00.610 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster1 2023-08-26T13:34:00.610 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T13:34:00.610 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=test 2023-08-26T13:34:00.611 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local state=started 2023-08-26T13:34:00.611 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local s 2023-08-26T13:34:00.611 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-08-26T13:34:00.611 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ sleep 1 2023-08-26T13:34:01.610 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ test_image_replay_state cluster1 mirror test started 2023-08-26T13:34:01.610 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster1 2023-08-26T13:34:01.610 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T13:34:01.611 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=test 2023-08-26T13:34:01.611 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local test_state=started 2023-08-26T13:34:01.611 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local status_result 2023-08-26T13:34:01.611 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local current_state=stopped 2023-08-26T13:34:01.615 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ admin_daemons cluster1 rbd mirror status mirror/test 2023-08-26T13:34:01.615 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local cluster_instance=cluster1 2023-08-26T13:34:01.615 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ shift 2023-08-26T13:34:01.615 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local cluster=cluster1 2023-08-26T13:34:01.615 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ grep -i state 2023-08-26T13:34:01.616 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local instance=cluster1 2023-08-26T13:34:01.616 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local loop_instance 2023-08-26T13:34:01.616 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-08-26T13:34:01.616 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ sleep 0 2023-08-26T13:34:01.616 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ '[' cluster1 '!=' cluster1 ']' 2023-08-26T13:34:01.617 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ seq 0 3 2023-08-26T13:34:01.617 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-08-26T13:34:01.617 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test 2023-08-26T13:34:01.618 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local cluster=cluster1:0 2023-08-26T13:34:01.618 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ shift 2023-08-26T13:34:01.618 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local instance 2023-08-26T13:34:01.618 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ set_cluster_instance cluster1:0 cluster instance 2023-08-26T13:34:01.618 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local val=cluster1:0 2023-08-26T13:34:01.619 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local cluster_var_name=cluster 2023-08-26T13:34:01.619 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local instance_var_name=instance 2023-08-26T13:34:01.619 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ cluster=cluster1 2023-08-26T13:34:01.619 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ instance=0 2023-08-26T13:34:01.619 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ '[' 0 = cluster1:0 ']' 2023-08-26T13:34:01.620 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ eval cluster=cluster1 2023-08-26T13:34:01.620 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ cluster=cluster1 2023-08-26T13:34:01.620 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ eval instance=0 2023-08-26T13:34:01.620 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ instance=0 2023-08-26T13:34:01.620 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ daemon_asok_file cluster1:0 cluster1 2023-08-26T13:34:01.620 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ local local_cluster=cluster1:0 2023-08-26T13:34:01.621 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ local cluster=cluster1 2023-08-26T13:34:01.621 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ local instance 2023-08-26T13:34:01.621 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2023-08-26T13:34:01.621 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ local val=cluster1:0 2023-08-26T13:34:01.621 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ local cluster_var_name=local_cluster 2023-08-26T13:34:01.622 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ local instance_var_name=instance 2023-08-26T13:34:01.622 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ cluster=cluster1 2023-08-26T13:34:01.622 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ instance=0 2023-08-26T13:34:01.622 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ '[' 0 = cluster1:0 ']' 2023-08-26T13:34:01.622 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ eval local_cluster=cluster1 2023-08-26T13:34:01.623 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++++ local_cluster=cluster1 2023-08-26T13:34:01.623 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ eval instance=0 2023-08-26T13:34:01.623 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++++ instance=0 2023-08-26T13:34:01.623 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2023-08-26T13:34:01.636 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-08-26T13:34:01.637 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-08-26T13:34:01.637 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-08-26T13:34:01.637 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2023-08-26T13:34:01.754 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ return 0 2023-08-26T13:34:01.755 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ status_result=' "state": "Replaying"' 2023-08-26T13:34:01.755 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ echo ' "state": "Replaying"' 2023-08-26T13:34:01.756 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ grep -i Replaying 2023-08-26T13:34:01.756 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: "state": "Replaying" 2023-08-26T13:34:01.759 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ current_state=started 2023-08-26T13:34:01.759 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ test started = started 2023-08-26T13:34:01.759 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ return 0 2023-08-26T13:34:01.759 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror test 2023-08-26T13:34:01.759 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local local_cluster=cluster1 2023-08-26T13:34:01.760 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster2 2023-08-26T13:34:01.760 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T13:34:01.760 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=test 2023-08-26T13:34:01.760 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ '[' snapshot = journal ']' 2023-08-26T13:34:01.760 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ '[' snapshot = snapshot ']' 2023-08-26T13:34:01.761 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror test 2023-08-26T13:34:01.761 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local local_cluster=cluster1 2023-08-26T13:34:01.761 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster2 2023-08-26T13:34:01.761 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T13:34:01.761 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=test 2023-08-26T13:34:01.762 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ mkfname cluster2-mirror-test.status 2023-08-26T13:34:01.762 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ echo cluster2-mirror-test.status 2023-08-26T13:34:01.762 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ sed -e 's|[/ ]|_|g' 2023-08-26T13:34:01.762 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local status_log=/tmp/tmp.TfDn6BcQ3N/cluster2-mirror-test.status 2023-08-26T13:34:01.762 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ mkfname cluster1-mirror-test.status 2023-08-26T13:34:01.763 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ echo cluster1-mirror-test.status 2023-08-26T13:34:01.763 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ sed -e 's|[/ ]|_|g' 2023-08-26T13:34:01.764 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local local_status_log=/tmp/tmp.TfDn6BcQ3N/cluster1-mirror-test.status 2023-08-26T13:34:01.764 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ mirror_image_snapshot cluster2 mirror test 2023-08-26T13:34:01.764 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster2 2023-08-26T13:34:01.764 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T13:34:01.765 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=test 2023-08-26T13:34:01.765 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/test 2023-08-26T13:34:02.614 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:Snapshot ID: 6 2023-08-26T13:34:02.621 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ get_newest_mirror_snapshot cluster2 mirror test /tmp/tmp.TfDn6BcQ3N/cluster2-mirror-test.status 2023-08-26T13:34:02.621 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster2 2023-08-26T13:34:02.622 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T13:34:02.622 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=test 2023-08-26T13:34:02.622 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local log=/tmp/tmp.TfDn6BcQ3N/cluster2-mirror-test.status 2023-08-26T13:34:02.622 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rbd --cluster cluster2 snap list --all mirror/test --format xml 2023-08-26T13:34:02.622 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-08-26T13:34:02.665 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ xmlstarlet sel -t -v //snapshot/id 2023-08-26T13:34:02.667 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local snapshot_id=6 2023-08-26T13:34:02.667 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ true 2023-08-26T13:34:02.668 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-08-26T13:34:02.668 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ sleep 0.2 2023-08-26T13:34:02.868 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.TfDn6BcQ3N/cluster1-mirror-test.status 2023-08-26T13:34:02.868 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster1 2023-08-26T13:34:02.869 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T13:34:02.869 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=test 2023-08-26T13:34:02.869 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local log=/tmp/tmp.TfDn6BcQ3N/cluster1-mirror-test.status 2023-08-26T13:34:02.869 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2023-08-26T13:34:02.870 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-08-26T13:34:02.915 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-08-26T13:34:02.917 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local primary_snapshot_id=4 2023-08-26T13:34:02.917 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ test 6 = 4 2023-08-26T13:34:02.917 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-08-26T13:34:02.917 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ sleep 0.4 2023-08-26T13:34:03.318 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.TfDn6BcQ3N/cluster1-mirror-test.status 2023-08-26T13:34:03.318 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster1 2023-08-26T13:34:03.319 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T13:34:03.319 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=test 2023-08-26T13:34:03.319 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local log=/tmp/tmp.TfDn6BcQ3N/cluster1-mirror-test.status 2023-08-26T13:34:03.319 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2023-08-26T13:34:03.320 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-08-26T13:34:03.365 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ true 2023-08-26T13:34:03.365 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-08-26T13:34:03.367 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:-:1.1: Document is empty 2023-08-26T13:34:03.368 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr: 2023-08-26T13:34:03.369 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:^ 2023-08-26T13:34:03.370 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local primary_snapshot_id= 2023-08-26T13:34:03.370 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ test 6 = '' 2023-08-26T13:34:03.371 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-08-26T13:34:03.371 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ sleep 0.8 2023-08-26T13:34:04.168 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.TfDn6BcQ3N/cluster1-mirror-test.status 2023-08-26T13:34:04.168 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster1 2023-08-26T13:34:04.168 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T13:34:04.168 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=test 2023-08-26T13:34:04.169 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local log=/tmp/tmp.TfDn6BcQ3N/cluster1-mirror-test.status 2023-08-26T13:34:04.169 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2023-08-26T13:34:04.169 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-08-26T13:34:04.233 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ true 2023-08-26T13:34:04.234 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-08-26T13:34:04.235 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:-:1.1: Document is empty 2023-08-26T13:34:04.235 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr: 2023-08-26T13:34:04.236 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:^ 2023-08-26T13:34:04.236 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local primary_snapshot_id= 2023-08-26T13:34:04.236 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ test 6 = '' 2023-08-26T13:34:04.236 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-08-26T13:34:04.236 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ sleep 1.6 2023-08-26T13:34:04.447 INFO:tasks.rbd_mirror_thrash:waiting for cluster2.client.mirror.0 2023-08-26T13:34:04.447 INFO:teuthology.orchestra.run:waiting for 600 2023-08-26T13:34:04.448 INFO:tasks.rbd_mirror_thrash:reviving cluster2.client.mirror.0 2023-08-26T13:34:04.448 INFO:tasks.rbd_mirror.cluster2.client.mirror.0:Restarting daemon 2023-08-26T13:34:04.448 DEBUG:teuthology.orchestra.run.smithi124:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.0 2023-08-26T13:34:04.451 INFO:tasks.rbd_mirror.cluster2.client.mirror.0:Started 2023-08-26T13:34:04.451 INFO:tasks.rbd_mirror_thrash:waiting for 81 secs before thrashing 2023-08-26T13:34:05.837 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.TfDn6BcQ3N/cluster1-mirror-test.status 2023-08-26T13:34:05.837 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster1 2023-08-26T13:34:05.837 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T13:34:05.838 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=test 2023-08-26T13:34:05.838 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local log=/tmp/tmp.TfDn6BcQ3N/cluster1-mirror-test.status 2023-08-26T13:34:05.838 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2023-08-26T13:34:05.838 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-08-26T13:34:05.885 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-08-26T13:34:05.887 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local primary_snapshot_id=6 2023-08-26T13:34:05.887 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ test 6 = 6 2023-08-26T13:34:05.888 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ return 0 2023-08-26T13:34:05.888 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ wait_for_snap_present cluster1 mirror test snap1 2023-08-26T13:34:05.888 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster1 2023-08-26T13:34:05.888 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T13:34:05.889 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=test 2023-08-26T13:34:05.889 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local snap_name=snap1 2023-08-26T13:34:05.889 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local s 2023-08-26T13:34:05.889 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2023-08-26T13:34:05.889 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ sleep 1 2023-08-26T13:34:06.888 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rbd --cluster cluster1 info mirror/test@snap1 2023-08-26T13:34:06.933 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:rbd image 'test': 2023-08-26T13:34:06.933 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: size 512 MiB in 128 objects 2023-08-26T13:34:06.934 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: order 22 (4 MiB objects) 2023-08-26T13:34:06.934 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: snapshot_count: 2 2023-08-26T13:34:06.934 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: id: 10ab632cdafc 2023-08-26T13:34:06.934 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: block_name_prefix: rbd_data.10ab632cdafc 2023-08-26T13:34:06.934 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: format: 2 2023-08-26T13:34:06.934 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: features: layering, exclusive-lock, non-primary 2023-08-26T13:34:06.935 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: op_features: 2023-08-26T13:34:06.935 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: flags: 2023-08-26T13:34:06.935 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: create_timestamp: Sat Aug 26 13:33:17 2023 2023-08-26T13:34:06.935 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: access_timestamp: Sat Aug 26 13:33:17 2023 2023-08-26T13:34:06.935 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: modify_timestamp: Sat Aug 26 13:33:17 2023 2023-08-26T13:34:06.936 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: protected: False 2023-08-26T13:34:06.936 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: mirroring state: enabled 2023-08-26T13:34:06.936 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: mirroring mode: snapshot 2023-08-26T13:34:06.936 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: mirroring global id: e2f2554d-659e-4467-b7f8-dddd7ccb0ee9 2023-08-26T13:34:06.936 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: mirroring primary: false 2023-08-26T13:34:06.938 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ return 0 2023-08-26T13:34:06.938 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ '[' -n '' ']' 2023-08-26T13:34:06.938 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ compare_image_snaps mirror test snap1 2023-08-26T13:34:06.938 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T13:34:06.938 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=test 2023-08-26T13:34:06.939 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local snap_name=snap1 2023-08-26T13:34:06.939 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local ret=0 2023-08-26T13:34:06.939 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local rmt_export=/tmp/tmp.TfDn6BcQ3N/cluster2-mirror-test.export 2023-08-26T13:34:06.939 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local loc_export=/tmp/tmp.TfDn6BcQ3N/cluster1-mirror-test.export 2023-08-26T13:34:06.939 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rm -f /tmp/tmp.TfDn6BcQ3N/cluster2-mirror-test.export /tmp/tmp.TfDn6BcQ3N/cluster1-mirror-test.export 2023-08-26T13:34:06.940 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rbd --cluster cluster2 -p mirror export test@snap1 /tmp/tmp.TfDn6BcQ3N/cluster2-mirror-test.export 2023-08-26T13:34:07.547 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2023-08-26T13:34:07.557 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rbd --cluster cluster1 -p mirror export test@snap1 /tmp/tmp.TfDn6BcQ3N/cluster1-mirror-test.export 2023-08-26T13:34:08.264 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2023-08-26T13:34:08.272 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ cmp /tmp/tmp.TfDn6BcQ3N/cluster2-mirror-test.export /tmp/tmp.TfDn6BcQ3N/cluster1-mirror-test.export 2023-08-26T13:34:08.484 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rm -f /tmp/tmp.TfDn6BcQ3N/cluster2-mirror-test.export /tmp/tmp.TfDn6BcQ3N/cluster1-mirror-test.export 2023-08-26T13:34:08.625 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ return 0 2023-08-26T13:34:08.625 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ clean_snap_name=snap1-clean 2023-08-26T13:34:08.625 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ create_snap cluster2 mirror test snap1-clean 2023-08-26T13:34:08.626 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster2 2023-08-26T13:34:08.626 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T13:34:08.626 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=test 2023-08-26T13:34:08.626 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local snap_name=snap1-clean 2023-08-26T13:34:08.626 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rbd --cluster cluster2 -p mirror snap create test@snap1-clean --debug-rbd=20 --debug-journaler=20 2023-08-26T13:34:09.532 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ for i in `seq 1 10` 2023-08-26T13:34:09.532 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ stress_write_image cluster2 mirror test 2023-08-26T13:34:09.532 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster2 2023-08-26T13:34:09.533 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T13:34:09.533 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=test 2023-08-26T13:34:09.533 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2023-08-26T13:34:09.534 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local duration=7 2023-08-26T13:34:09.534 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ set +e 2023-08-26T13:34:09.534 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ timeout 7s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror test --debug-rbd=20 --debug-journaler=20 2023-08-26T13:34:09.812 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:100 IOs, 57842688 bytes 2023-08-26T13:34:10.021 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:200 IOs, 120278016 bytes 2023-08-26T13:34:10.196 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:300 IOs, 186110976 bytes 2023-08-26T13:34:10.378 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:400 IOs, 231600128 bytes 2023-08-26T13:34:10.500 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:500 IOs, 263421952 bytes 2023-08-26T13:34:10.667 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:600 IOs, 307530752 bytes 2023-08-26T13:34:10.820 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:700 IOs, 356075520 bytes 2023-08-26T13:34:10.929 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:800 IOs, 380693504 bytes 2023-08-26T13:34:11.025 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:900 IOs, 395404288 bytes 2023-08-26T13:34:11.283 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:1000 IOs, 487259136 bytes 2023-08-26T13:34:11.449 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:1100 IOs, 556264448 bytes 2023-08-26T13:34:11.646 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:1200 IOs, 606843904 bytes 2023-08-26T13:34:11.952 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:1300 IOs, 725397504 bytes 2023-08-26T13:34:12.076 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:1400 IOs, 751455232 bytes 2023-08-26T13:34:12.186 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:1500 IOs, 771167232 bytes 2023-08-26T13:34:12.293 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:1600 IOs, 804635648 bytes 2023-08-26T13:34:12.394 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:1700 IOs, 812046336 bytes 2023-08-26T13:34:12.589 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:1800 IOs, 864214016 bytes 2023-08-26T13:34:12.765 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:1900 IOs, 928382976 bytes 2023-08-26T13:34:12.886 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:2000 IOs, 951983104 bytes 2023-08-26T13:34:13.124 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:2100 IOs, 1031437312 bytes 2023-08-26T13:34:13.295 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:2200 IOs, 1074677760 bytes 2023-08-26T13:34:13.490 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:2300 IOs, 1144961024 bytes 2023-08-26T13:34:13.665 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:2400 IOs, 1207606272 bytes 2023-08-26T13:34:13.818 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:2500 IOs, 1247447040 bytes 2023-08-26T13:34:13.990 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:2600 IOs, 1311710208 bytes 2023-08-26T13:34:14.162 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:2700 IOs, 1346437120 bytes 2023-08-26T13:34:14.259 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:2800 IOs, 1363152896 bytes 2023-08-26T13:34:14.402 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:2900 IOs, 1397406720 bytes 2023-08-26T13:34:14.514 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:3000 IOs, 1423063040 bytes 2023-08-26T13:34:14.646 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:3100 IOs, 1455142912 bytes 2023-08-26T13:34:14.808 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:3200 IOs, 1509562368 bytes 2023-08-26T13:34:14.877 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:3300 IOs, 1510189056 bytes 2023-08-26T13:34:14.955 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:3400 IOs, 1515079680 bytes 2023-08-26T13:34:15.085 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:3500 IOs, 1542073344 bytes 2023-08-26T13:34:15.150 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:3600 IOs, 1542723584 bytes 2023-08-26T13:34:15.253 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:3700 IOs, 1563357184 bytes 2023-08-26T13:34:15.386 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:3800 IOs, 1623059456 bytes 2023-08-26T13:34:15.541 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:3900 IOs, 1665266688 bytes 2023-08-26T13:34:15.724 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:4000 IOs, 1729678336 bytes 2023-08-26T13:34:15.937 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:4100 IOs, 1796542464 bytes 2023-08-26T13:34:16.059 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:4200 IOs, 1823856640 bytes 2023-08-26T13:34:16.228 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:4300 IOs, 1877993472 bytes 2023-08-26T13:34:16.388 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:4400 IOs, 1930142720 bytes 2023-08-26T13:34:16.539 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ error_code=124 2023-08-26T13:34:16.540 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ set -e 2023-08-26T13:34:16.540 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ '[' 124 -eq 124 ']' 2023-08-26T13:34:16.540 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ return 0 2023-08-26T13:34:16.541 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ wait_for_status_in_pool_dir cluster1 mirror test up+replaying 2023-08-26T13:34:16.541 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster1 2023-08-26T13:34:16.541 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T13:34:16.541 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=test 2023-08-26T13:34:16.542 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local state_pattern=up+replaying 2023-08-26T13:34:16.542 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local description_pattern= 2023-08-26T13:34:16.542 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local service_pattern= 2023-08-26T13:34:16.542 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-08-26T13:34:16.542 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ sleep 1 2023-08-26T13:34:17.540 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ test_status_in_pool_dir cluster1 mirror test up+replaying '' '' 2023-08-26T13:34:17.541 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster1 2023-08-26T13:34:17.541 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T13:34:17.541 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=test 2023-08-26T13:34:17.541 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local state_pattern=up+replaying 2023-08-26T13:34:17.541 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local description_pattern= 2023-08-26T13:34:17.542 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local service_pattern= 2023-08-26T13:34:17.542 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ mkfname cluster1-mirror-test.mirror_status 2023-08-26T13:34:17.542 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ echo cluster1-mirror-test.mirror_status 2023-08-26T13:34:17.542 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ sed -e 's|[/ ]|_|g' 2023-08-26T13:34:17.543 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local status_log=/tmp/tmp.TfDn6BcQ3N/cluster1-mirror-test.mirror_status 2023-08-26T13:34:17.543 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ CEPH_ARGS= 2023-08-26T13:34:17.543 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rbd --cluster cluster1 mirror image status mirror/test 2023-08-26T13:34:17.544 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ tee /tmp/tmp.TfDn6BcQ3N/cluster1-mirror-test.mirror_status 2023-08-26T13:34:17.591 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:test: 2023-08-26T13:34:17.591 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr: global_id: e2f2554d-659e-4467-b7f8-dddd7ccb0ee9 2023-08-26T13:34:17.591 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr: state: up+replaying 2023-08-26T13:34:17.592 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr: description: replaying, {"bytes_per_second":17844701.87,"bytes_per_snapshot":267670528.0,"last_snapshot_bytes":535341056,"last_snapshot_sync_seconds":1,"local_snapshot_timestamp":1693056842,"remote_snapshot_timestamp":1693056842,"replay_state":"idle"} 2023-08-26T13:34:17.592 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr: service: mirror.0 on smithi124 2023-08-26T13:34:17.592 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr: last_update: 2023-08-26 13:34:15 2023-08-26T13:34:17.592 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr: peer_sites: 2023-08-26T13:34:17.592 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr: name: cluster2 2023-08-26T13:34:17.592 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr: state: down+unknown 2023-08-26T13:34:17.593 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr: description: status not found 2023-08-26T13:34:17.593 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr: last_update: 2023-08-26T13:34:17.596 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ grep '^ state: .*up+replaying' /tmp/tmp.TfDn6BcQ3N/cluster1-mirror-test.mirror_status 2023-08-26T13:34:17.597 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: state: up+replaying 2023-08-26T13:34:17.597 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ grep '^ description: .*' /tmp/tmp.TfDn6BcQ3N/cluster1-mirror-test.mirror_status 2023-08-26T13:34:17.598 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: description: replaying, {"bytes_per_second":17844701.87,"bytes_per_snapshot":267670528.0,"last_snapshot_bytes":535341056,"last_snapshot_sync_seconds":1,"local_snapshot_timestamp":1693056842,"remote_snapshot_timestamp":1693056842,"replay_state":"idle"} 2023-08-26T13:34:17.598 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ '[' -n '' ']' 2023-08-26T13:34:17.599 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ echo up+replaying 2023-08-26T13:34:17.599 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ grep '^up+' 2023-08-26T13:34:17.599 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:up+replaying 2023-08-26T13:34:17.600 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ grep 'service: *mirror..* on ' /tmp/tmp.TfDn6BcQ3N/cluster1-mirror-test.mirror_status 2023-08-26T13:34:17.600 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: service: mirror.0 on smithi124 2023-08-26T13:34:17.600 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' /tmp/tmp.TfDn6BcQ3N/cluster1-mirror-test.mirror_status 2023-08-26T13:34:17.602 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local 'last_update=2023-08-26 13:34:15' 2023-08-26T13:34:17.602 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ test_mirror_pool_status_verbose cluster1 mirror test up+replaying '2023-08-26 13:34:15' 2023-08-26T13:34:17.602 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster1 2023-08-26T13:34:17.602 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T13:34:17.603 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=test 2023-08-26T13:34:17.603 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local state_pattern=up+replaying 2023-08-26T13:34:17.603 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local 'prev_last_update=2023-08-26 13:34:15' 2023-08-26T13:34:17.603 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ mkfname cluster1-mirror.mirror_status 2023-08-26T13:34:17.603 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ echo cluster1-mirror.mirror_status 2023-08-26T13:34:17.604 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ sed -e 's|[/ ]|_|g' 2023-08-26T13:34:17.604 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local status_log=/tmp/tmp.TfDn6BcQ3N/cluster1-mirror.mirror_status 2023-08-26T13:34:17.605 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2023-08-26T13:34:17.640 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:rbd: failed to query services: (13) Permission denied 2023-08-26T13:34:17.661 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local last_update state 2023-08-26T13:34:17.661 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update' 2023-08-26T13:34:17.663 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ last_update='2023-08-26 13:34:15' 2023-08-26T13:34:17.663 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state' 2023-08-26T13:34:17.665 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ state=up+replaying 2023-08-26T13:34:17.665 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ echo up+replaying 2023-08-26T13:34:17.665 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ grep up+replaying 2023-08-26T13:34:17.666 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:up+replaying 2023-08-26T13:34:17.667 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ return 0 2023-08-26T13:34:17.667 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ return 0 2023-08-26T13:34:17.667 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ snap_name=snap2 2023-08-26T13:34:17.668 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ create_snap cluster2 mirror test snap2 2023-08-26T13:34:17.668 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster2 2023-08-26T13:34:17.668 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T13:34:17.668 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=test 2023-08-26T13:34:17.668 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local snap_name=snap2 2023-08-26T13:34:17.669 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rbd --cluster cluster2 -p mirror snap create test@snap2 --debug-rbd=20 --debug-journaler=20 2023-08-26T13:34:46.580 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ wait_for_image_replay_started cluster1 mirror test 2023-08-26T13:34:46.580 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster1 2023-08-26T13:34:46.580 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T13:34:46.581 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=test 2023-08-26T13:34:46.581 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ wait_for_image_replay_state cluster1 mirror test started 2023-08-26T13:34:46.581 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster1 2023-08-26T13:34:46.581 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T13:34:46.581 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=test 2023-08-26T13:34:46.582 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local state=started 2023-08-26T13:34:46.582 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local s 2023-08-26T13:34:46.582 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-08-26T13:34:46.582 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ sleep 1 2023-08-26T13:34:47.582 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ test_image_replay_state cluster1 mirror test started 2023-08-26T13:34:47.583 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster1 2023-08-26T13:34:47.583 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T13:34:47.583 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=test 2023-08-26T13:34:47.583 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local test_state=started 2023-08-26T13:34:47.583 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local status_result 2023-08-26T13:34:47.584 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local current_state=stopped 2023-08-26T13:34:47.584 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ admin_daemons cluster1 rbd mirror status mirror/test 2023-08-26T13:34:47.584 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ grep -i state 2023-08-26T13:34:47.584 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local cluster_instance=cluster1 2023-08-26T13:34:47.585 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ shift 2023-08-26T13:34:47.585 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local cluster=cluster1 2023-08-26T13:34:47.585 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local instance=cluster1 2023-08-26T13:34:47.585 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local loop_instance 2023-08-26T13:34:47.586 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-08-26T13:34:47.586 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ sleep 0 2023-08-26T13:34:47.589 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ '[' cluster1 '!=' cluster1 ']' 2023-08-26T13:34:47.590 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ seq 0 3 2023-08-26T13:34:47.590 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-08-26T13:34:47.590 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test 2023-08-26T13:34:47.590 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local cluster=cluster1:0 2023-08-26T13:34:47.590 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ shift 2023-08-26T13:34:47.591 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local instance 2023-08-26T13:34:47.591 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ set_cluster_instance cluster1:0 cluster instance 2023-08-26T13:34:47.591 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local val=cluster1:0 2023-08-26T13:34:47.591 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local cluster_var_name=cluster 2023-08-26T13:34:47.591 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local instance_var_name=instance 2023-08-26T13:34:47.592 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ cluster=cluster1 2023-08-26T13:34:47.592 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ instance=0 2023-08-26T13:34:47.592 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ '[' 0 = cluster1:0 ']' 2023-08-26T13:34:47.592 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ eval cluster=cluster1 2023-08-26T13:34:47.593 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ cluster=cluster1 2023-08-26T13:34:47.593 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ eval instance=0 2023-08-26T13:34:47.593 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ instance=0 2023-08-26T13:34:47.593 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ daemon_asok_file cluster1:0 cluster1 2023-08-26T13:34:47.593 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ local local_cluster=cluster1:0 2023-08-26T13:34:47.594 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ local cluster=cluster1 2023-08-26T13:34:47.594 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ local instance 2023-08-26T13:34:47.594 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2023-08-26T13:34:47.594 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ local val=cluster1:0 2023-08-26T13:34:47.594 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ local cluster_var_name=local_cluster 2023-08-26T13:34:47.594 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ local instance_var_name=instance 2023-08-26T13:34:47.595 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ cluster=cluster1 2023-08-26T13:34:47.595 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ instance=0 2023-08-26T13:34:47.595 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ '[' 0 = cluster1:0 ']' 2023-08-26T13:34:47.595 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ eval local_cluster=cluster1 2023-08-26T13:34:47.596 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++++ local_cluster=cluster1 2023-08-26T13:34:47.596 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ eval instance=0 2023-08-26T13:34:47.596 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++++ instance=0 2023-08-26T13:34:47.596 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2023-08-26T13:34:47.608 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-08-26T13:34:47.608 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-08-26T13:34:47.609 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-08-26T13:34:47.609 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2023-08-26T13:34:47.723 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ return 0 2023-08-26T13:34:47.723 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ status_result=' "state": "Replaying"' 2023-08-26T13:34:47.724 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ echo ' "state": "Replaying"' 2023-08-26T13:34:47.724 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ grep -i Replaying 2023-08-26T13:34:47.724 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: "state": "Replaying" 2023-08-26T13:34:47.727 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ current_state=started 2023-08-26T13:34:47.727 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ test started = started 2023-08-26T13:34:47.727 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ return 0 2023-08-26T13:34:47.728 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror test 2023-08-26T13:34:47.728 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local local_cluster=cluster1 2023-08-26T13:34:47.728 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster2 2023-08-26T13:34:47.728 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T13:34:47.728 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=test 2023-08-26T13:34:47.729 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ '[' snapshot = journal ']' 2023-08-26T13:34:47.729 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ '[' snapshot = snapshot ']' 2023-08-26T13:34:47.729 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror test 2023-08-26T13:34:47.729 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local local_cluster=cluster1 2023-08-26T13:34:47.729 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster2 2023-08-26T13:34:47.730 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T13:34:47.730 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=test 2023-08-26T13:34:47.730 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ mkfname cluster2-mirror-test.status 2023-08-26T13:34:47.730 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ echo cluster2-mirror-test.status 2023-08-26T13:34:47.730 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ sed -e 's|[/ ]|_|g' 2023-08-26T13:34:47.731 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local status_log=/tmp/tmp.TfDn6BcQ3N/cluster2-mirror-test.status 2023-08-26T13:34:47.731 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ mkfname cluster1-mirror-test.status 2023-08-26T13:34:47.731 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ echo cluster1-mirror-test.status 2023-08-26T13:34:47.731 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ sed -e 's|[/ ]|_|g' 2023-08-26T13:34:47.732 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local local_status_log=/tmp/tmp.TfDn6BcQ3N/cluster1-mirror-test.status 2023-08-26T13:34:47.732 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ mirror_image_snapshot cluster2 mirror test 2023-08-26T13:34:47.732 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster2 2023-08-26T13:34:47.733 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T13:34:47.733 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=test 2023-08-26T13:34:47.733 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/test 2023-08-26T13:34:49.581 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:Snapshot ID: 9 2023-08-26T13:34:49.592 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ get_newest_mirror_snapshot cluster2 mirror test /tmp/tmp.TfDn6BcQ3N/cluster2-mirror-test.status 2023-08-26T13:34:49.593 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster2 2023-08-26T13:34:49.593 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T13:34:49.593 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=test 2023-08-26T13:34:49.593 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local log=/tmp/tmp.TfDn6BcQ3N/cluster2-mirror-test.status 2023-08-26T13:34:49.593 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rbd --cluster cluster2 snap list --all mirror/test --format xml 2023-08-26T13:34:49.594 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-08-26T13:34:49.640 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ xmlstarlet sel -t -v //snapshot/id 2023-08-26T13:34:49.642 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local snapshot_id=9 2023-08-26T13:34:49.642 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ true 2023-08-26T13:34:49.643 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-08-26T13:34:49.643 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ sleep 0.2 2023-08-26T13:34:49.844 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.TfDn6BcQ3N/cluster1-mirror-test.status 2023-08-26T13:34:49.844 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster1 2023-08-26T13:34:49.844 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T13:34:49.844 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=test 2023-08-26T13:34:49.845 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local log=/tmp/tmp.TfDn6BcQ3N/cluster1-mirror-test.status 2023-08-26T13:34:49.845 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2023-08-26T13:34:49.845 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-08-26T13:34:49.892 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ true 2023-08-26T13:34:49.892 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-08-26T13:34:49.893 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:-:1.1: Document is empty 2023-08-26T13:34:49.893 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr: 2023-08-26T13:34:49.894 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:^ 2023-08-26T13:34:49.894 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local primary_snapshot_id= 2023-08-26T13:34:49.895 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ test 9 = '' 2023-08-26T13:34:49.895 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-08-26T13:34:49.895 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ sleep 0.4 2023-08-26T13:34:50.295 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.TfDn6BcQ3N/cluster1-mirror-test.status 2023-08-26T13:34:50.296 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster1 2023-08-26T13:34:50.296 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T13:34:50.296 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=test 2023-08-26T13:34:50.297 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local log=/tmp/tmp.TfDn6BcQ3N/cluster1-mirror-test.status 2023-08-26T13:34:50.297 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2023-08-26T13:34:50.297 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-08-26T13:34:50.346 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ true 2023-08-26T13:34:50.347 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-08-26T13:34:50.348 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:-:1.1: Document is empty 2023-08-26T13:34:50.349 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr: 2023-08-26T13:34:50.349 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:^ 2023-08-26T13:34:50.349 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local primary_snapshot_id= 2023-08-26T13:34:50.349 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ test 9 = '' 2023-08-26T13:34:50.349 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-08-26T13:34:50.350 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ sleep 0.8 2023-08-26T13:34:51.150 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.TfDn6BcQ3N/cluster1-mirror-test.status 2023-08-26T13:34:51.150 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster1 2023-08-26T13:34:51.151 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T13:34:51.151 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=test 2023-08-26T13:34:51.151 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local log=/tmp/tmp.TfDn6BcQ3N/cluster1-mirror-test.status 2023-08-26T13:34:51.151 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2023-08-26T13:34:51.152 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-08-26T13:34:51.228 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ true 2023-08-26T13:34:51.228 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-08-26T13:34:51.229 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:-:1.1: Document is empty 2023-08-26T13:34:51.229 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr: 2023-08-26T13:34:51.230 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:^ 2023-08-26T13:34:51.230 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local primary_snapshot_id= 2023-08-26T13:34:51.231 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ test 9 = '' 2023-08-26T13:34:51.231 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-08-26T13:34:51.231 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ sleep 1.6 2023-08-26T13:34:52.831 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.TfDn6BcQ3N/cluster1-mirror-test.status 2023-08-26T13:34:52.832 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster1 2023-08-26T13:34:52.832 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T13:34:52.832 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=test 2023-08-26T13:34:52.832 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local log=/tmp/tmp.TfDn6BcQ3N/cluster1-mirror-test.status 2023-08-26T13:34:52.833 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2023-08-26T13:34:52.833 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-08-26T13:34:52.882 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-08-26T13:34:52.885 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local primary_snapshot_id=9 2023-08-26T13:34:52.885 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ test 9 = 9 2023-08-26T13:34:52.885 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ return 0 2023-08-26T13:34:52.885 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ wait_for_snap_present cluster1 mirror test snap2 2023-08-26T13:34:52.885 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster1 2023-08-26T13:34:52.886 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T13:34:52.886 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=test 2023-08-26T13:34:52.886 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local snap_name=snap2 2023-08-26T13:34:52.886 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local s 2023-08-26T13:34:52.886 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2023-08-26T13:34:52.886 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ sleep 1 2023-08-26T13:34:53.885 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rbd --cluster cluster1 info mirror/test@snap2 2023-08-26T13:34:53.932 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:rbd image 'test': 2023-08-26T13:34:53.932 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: size 512 MiB in 128 objects 2023-08-26T13:34:53.933 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: order 22 (4 MiB objects) 2023-08-26T13:34:53.933 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: snapshot_count: 4 2023-08-26T13:34:53.933 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: id: 10ab632cdafc 2023-08-26T13:34:53.933 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: block_name_prefix: rbd_data.10ab632cdafc 2023-08-26T13:34:53.933 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: format: 2 2023-08-26T13:34:53.933 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: features: layering, exclusive-lock, non-primary 2023-08-26T13:34:53.934 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: op_features: 2023-08-26T13:34:53.934 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: flags: 2023-08-26T13:34:53.934 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: create_timestamp: Sat Aug 26 13:33:17 2023 2023-08-26T13:34:53.934 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: access_timestamp: Sat Aug 26 13:33:17 2023 2023-08-26T13:34:53.934 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: modify_timestamp: Sat Aug 26 13:33:17 2023 2023-08-26T13:34:53.935 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: protected: False 2023-08-26T13:34:53.935 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: mirroring state: enabled 2023-08-26T13:34:53.935 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: mirroring mode: snapshot 2023-08-26T13:34:53.935 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: mirroring global id: e2f2554d-659e-4467-b7f8-dddd7ccb0ee9 2023-08-26T13:34:53.935 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: mirroring primary: false 2023-08-26T13:34:53.937 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ return 0 2023-08-26T13:34:53.937 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ '[' -n snap1-clean ']' 2023-08-26T13:34:53.937 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ compare_image_snaps mirror test snap1-clean 2023-08-26T13:34:53.937 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T13:34:53.938 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=test 2023-08-26T13:34:53.938 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local snap_name=snap1-clean 2023-08-26T13:34:53.938 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local ret=0 2023-08-26T13:34:53.938 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local rmt_export=/tmp/tmp.TfDn6BcQ3N/cluster2-mirror-test.export 2023-08-26T13:34:53.938 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local loc_export=/tmp/tmp.TfDn6BcQ3N/cluster1-mirror-test.export 2023-08-26T13:34:53.938 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rm -f /tmp/tmp.TfDn6BcQ3N/cluster2-mirror-test.export /tmp/tmp.TfDn6BcQ3N/cluster1-mirror-test.export 2023-08-26T13:34:53.939 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rbd --cluster cluster2 -p mirror export test@snap1-clean /tmp/tmp.TfDn6BcQ3N/cluster2-mirror-test.export 2023-08-26T13:34:54.521 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2023-08-26T13:34:54.531 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rbd --cluster cluster1 -p mirror export test@snap1-clean /tmp/tmp.TfDn6BcQ3N/cluster1-mirror-test.export 2023-08-26T13:34:55.249 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2023-08-26T13:34:55.259 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ cmp /tmp/tmp.TfDn6BcQ3N/cluster2-mirror-test.export /tmp/tmp.TfDn6BcQ3N/cluster1-mirror-test.export 2023-08-26T13:34:55.473 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rm -f /tmp/tmp.TfDn6BcQ3N/cluster2-mirror-test.export /tmp/tmp.TfDn6BcQ3N/cluster1-mirror-test.export 2023-08-26T13:34:55.618 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ return 0 2023-08-26T13:34:55.619 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ compare_image_snaps mirror test snap2 2023-08-26T13:34:55.619 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T13:34:55.619 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=test 2023-08-26T13:34:55.619 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local snap_name=snap2 2023-08-26T13:34:55.619 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local ret=0 2023-08-26T13:34:55.620 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local rmt_export=/tmp/tmp.TfDn6BcQ3N/cluster2-mirror-test.export 2023-08-26T13:34:55.620 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local loc_export=/tmp/tmp.TfDn6BcQ3N/cluster1-mirror-test.export 2023-08-26T13:34:55.620 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rm -f /tmp/tmp.TfDn6BcQ3N/cluster2-mirror-test.export /tmp/tmp.TfDn6BcQ3N/cluster1-mirror-test.export 2023-08-26T13:34:55.620 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rbd --cluster cluster2 -p mirror export test@snap2 /tmp/tmp.TfDn6BcQ3N/cluster2-mirror-test.export 2023-08-26T13:34:56.206 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2023-08-26T13:34:56.215 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rbd --cluster cluster1 -p mirror export test@snap2 /tmp/tmp.TfDn6BcQ3N/cluster1-mirror-test.export 2023-08-26T13:34:56.886 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2023-08-26T13:34:56.894 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ cmp /tmp/tmp.TfDn6BcQ3N/cluster2-mirror-test.export /tmp/tmp.TfDn6BcQ3N/cluster1-mirror-test.export 2023-08-26T13:34:57.103 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rm -f /tmp/tmp.TfDn6BcQ3N/cluster2-mirror-test.export /tmp/tmp.TfDn6BcQ3N/cluster1-mirror-test.export 2023-08-26T13:34:57.246 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ return 0 2023-08-26T13:34:57.246 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ clean_snap_name=snap2-clean 2023-08-26T13:34:57.247 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ create_snap cluster2 mirror test snap2-clean 2023-08-26T13:34:57.247 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster2 2023-08-26T13:34:57.247 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T13:34:57.247 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=test 2023-08-26T13:34:57.247 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local snap_name=snap2-clean 2023-08-26T13:34:57.247 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rbd --cluster cluster2 -p mirror snap create test@snap2-clean --debug-rbd=20 --debug-journaler=20 2023-08-26T13:34:57.595 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ for i in `seq 1 10` 2023-08-26T13:34:57.595 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ stress_write_image cluster2 mirror test 2023-08-26T13:34:57.595 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster2 2023-08-26T13:34:57.596 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T13:34:57.596 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=test 2023-08-26T13:34:57.596 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2023-08-26T13:34:57.597 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local duration=8 2023-08-26T13:34:57.597 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ set +e 2023-08-26T13:34:57.597 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ timeout 8s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror test --debug-rbd=20 --debug-journaler=20 2023-08-26T13:34:57.859 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:100 IOs, 55069696 bytes 2023-08-26T13:34:58.040 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:200 IOs, 107639808 bytes 2023-08-26T13:34:58.283 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:300 IOs, 208484352 bytes 2023-08-26T13:34:58.399 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:400 IOs, 221168640 bytes 2023-08-26T13:34:58.609 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:500 IOs, 296591360 bytes 2023-08-26T13:34:58.764 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:600 IOs, 346169344 bytes 2023-08-26T13:34:58.867 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:700 IOs, 366186496 bytes 2023-08-26T13:34:59.038 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:800 IOs, 422223872 bytes 2023-08-26T13:34:59.141 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:900 IOs, 440451072 bytes 2023-08-26T13:34:59.317 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:1000 IOs, 490238976 bytes 2023-08-26T13:34:59.691 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:1100 IOs, 641642496 bytes 2023-08-26T13:34:59.816 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:1200 IOs, 671835136 bytes 2023-08-26T13:34:59.960 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:1300 IOs, 703946752 bytes 2023-08-26T13:35:00.097 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:1400 IOs, 746975232 bytes 2023-08-26T13:35:00.282 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:1500 IOs, 808054784 bytes 2023-08-26T13:35:00.447 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:1600 IOs, 847599616 bytes 2023-08-26T13:35:00.657 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:1700 IOs, 935105536 bytes 2023-08-26T13:35:00.928 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:1800 IOs, 1017234432 bytes 2023-08-26T13:35:01.106 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:1900 IOs, 1073245184 bytes 2023-08-26T13:35:01.244 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:2000 IOs, 1116259328 bytes 2023-08-26T13:35:01.364 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:2100 IOs, 1140073472 bytes 2023-08-26T13:35:01.445 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:2200 IOs, 1151174656 bytes 2023-08-26T13:35:01.548 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:2300 IOs, 1161894912 bytes 2023-08-26T13:35:01.870 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:2400 IOs, 1302301696 bytes 2023-08-26T13:35:01.961 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:2500 IOs, 1343385600 bytes 2023-08-26T13:35:02.251 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:2600 IOs, 1430716416 bytes 2023-08-26T13:35:02.605 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:2700 IOs, 1483649024 bytes 2023-08-26T13:35:02.771 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:2800 IOs, 1529937920 bytes 2023-08-26T13:35:02.981 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:2900 IOs, 1597331456 bytes 2023-08-26T13:35:03.167 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:3000 IOs, 1651261440 bytes 2023-08-26T13:35:03.375 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:3100 IOs, 1727531008 bytes 2023-08-26T13:35:03.472 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:3200 IOs, 1747641344 bytes 2023-08-26T13:35:03.644 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:3300 IOs, 1808516096 bytes 2023-08-26T13:35:03.789 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:3400 IOs, 1851406336 bytes 2023-08-26T13:35:03.912 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:3500 IOs, 1881138176 bytes 2023-08-26T13:35:04.035 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:3600 IOs, 1912261632 bytes 2023-08-26T13:35:04.254 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:3700 IOs, 1991654400 bytes 2023-08-26T13:35:04.416 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:3800 IOs, 2039716864 bytes 2023-08-26T13:35:04.602 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:3900 IOs, 2095492096 bytes 2023-08-26T13:35:04.685 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:4000 IOs, 2100842496 bytes 2023-08-26T13:35:04.791 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:4100 IOs, 2119299072 bytes 2023-08-26T13:35:04.919 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:4200 IOs, 2144695296 bytes 2023-08-26T13:35:05.141 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:4300 IOs, 2214917120 bytes 2023-08-26T13:35:05.282 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:4400 IOs, 2264983552 bytes 2023-08-26T13:35:05.545 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:4500 IOs, 2352601088 bytes 2023-08-26T13:35:05.601 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ error_code=124 2023-08-26T13:35:05.601 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ set -e 2023-08-26T13:35:05.601 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ '[' 124 -eq 124 ']' 2023-08-26T13:35:05.602 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ return 0 2023-08-26T13:35:05.602 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ wait_for_status_in_pool_dir cluster1 mirror test up+replaying 2023-08-26T13:35:05.602 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster1 2023-08-26T13:35:05.602 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T13:35:05.602 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=test 2023-08-26T13:35:05.603 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local state_pattern=up+replaying 2023-08-26T13:35:05.603 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local description_pattern= 2023-08-26T13:35:05.603 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local service_pattern= 2023-08-26T13:35:05.603 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-08-26T13:35:05.603 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ sleep 1 2023-08-26T13:35:06.452 INFO:tasks.rbd_mirror_thrash:kill cluster1.client.mirror.0 2023-08-26T13:35:06.453 INFO:tasks.rbd_mirror.cluster1.client.mirror.0:Sent signal 15 2023-08-26T13:35:06.453 INFO:tasks.rbd_mirror_thrash:waiting for 3 secs before reviving daemons 2023-08-26T13:35:06.453 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi124.stderr:2023-08-26T13:35:06.453+0000 7fec126e6700 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.0 (PID: 105952) UID: 1000 2023-08-26T13:35:06.603 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ test_status_in_pool_dir cluster1 mirror test up+replaying '' '' 2023-08-26T13:35:06.603 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster1 2023-08-26T13:35:06.603 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T13:35:06.603 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=test 2023-08-26T13:35:06.604 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local state_pattern=up+replaying 2023-08-26T13:35:06.604 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local description_pattern= 2023-08-26T13:35:06.604 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local service_pattern= 2023-08-26T13:35:06.604 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ mkfname cluster1-mirror-test.mirror_status 2023-08-26T13:35:06.604 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ echo cluster1-mirror-test.mirror_status 2023-08-26T13:35:06.605 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ sed -e 's|[/ ]|_|g' 2023-08-26T13:35:06.605 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local status_log=/tmp/tmp.TfDn6BcQ3N/cluster1-mirror-test.mirror_status 2023-08-26T13:35:06.605 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ CEPH_ARGS= 2023-08-26T13:35:06.606 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rbd --cluster cluster1 mirror image status mirror/test 2023-08-26T13:35:06.606 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ tee /tmp/tmp.TfDn6BcQ3N/cluster1-mirror-test.mirror_status 2023-08-26T13:35:06.647 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:test: 2023-08-26T13:35:06.648 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr: global_id: e2f2554d-659e-4467-b7f8-dddd7ccb0ee9 2023-08-26T13:35:06.648 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr: state: down+stopped 2023-08-26T13:35:06.648 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr: description: stopped 2023-08-26T13:35:06.648 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr: last_update: 2023-08-26 13:35:06 2023-08-26T13:35:06.649 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr: peer_sites: 2023-08-26T13:35:06.649 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr: name: cluster2 2023-08-26T13:35:06.649 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr: state: up+stopped 2023-08-26T13:35:06.649 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr: description: local image is primary 2023-08-26T13:35:06.649 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr: last_update: 2023-08-26 13:34:45 2023-08-26T13:35:06.652 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ grep '^ state: .*up+replaying' /tmp/tmp.TfDn6BcQ3N/cluster1-mirror-test.mirror_status 2023-08-26T13:35:06.653 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ return 1 2023-08-26T13:35:06.653 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-08-26T13:35:06.653 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ sleep 2 2023-08-26T13:35:08.654 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ test_status_in_pool_dir cluster1 mirror test up+replaying '' '' 2023-08-26T13:35:08.655 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster1 2023-08-26T13:35:08.655 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T13:35:08.655 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=test 2023-08-26T13:35:08.655 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local state_pattern=up+replaying 2023-08-26T13:35:08.656 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local description_pattern= 2023-08-26T13:35:08.656 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local service_pattern= 2023-08-26T13:35:08.656 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ mkfname cluster1-mirror-test.mirror_status 2023-08-26T13:35:08.657 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ echo cluster1-mirror-test.mirror_status 2023-08-26T13:35:08.657 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ sed -e 's|[/ ]|_|g' 2023-08-26T13:35:08.657 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local status_log=/tmp/tmp.TfDn6BcQ3N/cluster1-mirror-test.mirror_status 2023-08-26T13:35:08.658 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ CEPH_ARGS= 2023-08-26T13:35:08.658 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rbd --cluster cluster1 mirror image status mirror/test 2023-08-26T13:35:08.658 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ tee /tmp/tmp.TfDn6BcQ3N/cluster1-mirror-test.mirror_status 2023-08-26T13:35:08.706 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:test: 2023-08-26T13:35:08.706 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr: global_id: e2f2554d-659e-4467-b7f8-dddd7ccb0ee9 2023-08-26T13:35:08.706 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr: state: down+stopped 2023-08-26T13:35:08.706 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr: description: stopped 2023-08-26T13:35:08.707 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr: last_update: 2023-08-26 13:35:06 2023-08-26T13:35:08.707 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr: peer_sites: 2023-08-26T13:35:08.707 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr: name: cluster2 2023-08-26T13:35:08.707 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr: state: up+stopped 2023-08-26T13:35:08.707 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr: description: local image is primary 2023-08-26T13:35:08.707 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr: last_update: 2023-08-26 13:34:45 2023-08-26T13:35:08.711 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ grep '^ state: .*up+replaying' /tmp/tmp.TfDn6BcQ3N/cluster1-mirror-test.mirror_status 2023-08-26T13:35:08.712 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ return 1 2023-08-26T13:35:08.712 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-08-26T13:35:08.712 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ sleep 4 2023-08-26T13:35:09.453 INFO:tasks.rbd_mirror_thrash:waiting for cluster1.client.mirror.0 2023-08-26T13:35:09.454 INFO:teuthology.orchestra.run:waiting for 600 2023-08-26T13:35:09.454 INFO:tasks.rbd_mirror_thrash:reviving cluster1.client.mirror.0 2023-08-26T13:35:09.454 INFO:tasks.rbd_mirror.cluster1.client.mirror.0:Restarting daemon 2023-08-26T13:35:09.455 DEBUG:teuthology.orchestra.run.smithi124:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.0 2023-08-26T13:35:09.457 INFO:tasks.rbd_mirror.cluster1.client.mirror.0:Started 2023-08-26T13:35:09.457 INFO:tasks.rbd_mirror_thrash:waiting for 76 secs before thrashing 2023-08-26T13:35:12.714 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ test_status_in_pool_dir cluster1 mirror test up+replaying '' '' 2023-08-26T13:35:12.714 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster1 2023-08-26T13:35:12.714 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T13:35:12.714 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=test 2023-08-26T13:35:12.715 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local state_pattern=up+replaying 2023-08-26T13:35:12.715 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local description_pattern= 2023-08-26T13:35:12.715 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local service_pattern= 2023-08-26T13:35:12.715 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ mkfname cluster1-mirror-test.mirror_status 2023-08-26T13:35:12.715 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ echo cluster1-mirror-test.mirror_status 2023-08-26T13:35:12.715 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ sed -e 's|[/ ]|_|g' 2023-08-26T13:35:12.716 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local status_log=/tmp/tmp.TfDn6BcQ3N/cluster1-mirror-test.mirror_status 2023-08-26T13:35:12.716 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ CEPH_ARGS= 2023-08-26T13:35:12.717 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rbd --cluster cluster1 mirror image status mirror/test 2023-08-26T13:35:12.717 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ tee /tmp/tmp.TfDn6BcQ3N/cluster1-mirror-test.mirror_status 2023-08-26T13:35:12.764 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:test: 2023-08-26T13:35:12.764 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr: global_id: e2f2554d-659e-4467-b7f8-dddd7ccb0ee9 2023-08-26T13:35:12.764 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr: state: down+unknown 2023-08-26T13:35:12.765 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr: description: status not found 2023-08-26T13:35:12.765 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr: last_update: 2023-08-26T13:35:12.765 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr: peer_sites: 2023-08-26T13:35:12.765 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr: name: cluster2 2023-08-26T13:35:12.766 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr: state: up+stopped 2023-08-26T13:35:12.766 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr: description: local image is primary 2023-08-26T13:35:12.766 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr: last_update: 2023-08-26 13:34:45 2023-08-26T13:35:12.770 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ grep '^ state: .*up+replaying' /tmp/tmp.TfDn6BcQ3N/cluster1-mirror-test.mirror_status 2023-08-26T13:35:12.771 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ return 1 2023-08-26T13:35:12.771 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-08-26T13:35:12.771 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ sleep 8 2023-08-26T13:35:20.773 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ test_status_in_pool_dir cluster1 mirror test up+replaying '' '' 2023-08-26T13:35:20.773 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster1 2023-08-26T13:35:20.773 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T13:35:20.774 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=test 2023-08-26T13:35:20.774 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local state_pattern=up+replaying 2023-08-26T13:35:20.774 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local description_pattern= 2023-08-26T13:35:20.774 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local service_pattern= 2023-08-26T13:35:20.774 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ mkfname cluster1-mirror-test.mirror_status 2023-08-26T13:35:20.775 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ echo cluster1-mirror-test.mirror_status 2023-08-26T13:35:20.775 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ sed -e 's|[/ ]|_|g' 2023-08-26T13:35:20.775 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local status_log=/tmp/tmp.TfDn6BcQ3N/cluster1-mirror-test.mirror_status 2023-08-26T13:35:20.776 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ CEPH_ARGS= 2023-08-26T13:35:20.776 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rbd --cluster cluster1 mirror image status mirror/test 2023-08-26T13:35:20.776 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ tee /tmp/tmp.TfDn6BcQ3N/cluster1-mirror-test.mirror_status 2023-08-26T13:35:20.823 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:test: 2023-08-26T13:35:20.823 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr: global_id: e2f2554d-659e-4467-b7f8-dddd7ccb0ee9 2023-08-26T13:35:20.823 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr: state: down+unknown 2023-08-26T13:35:20.823 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr: description: status not found 2023-08-26T13:35:20.824 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr: last_update: 2023-08-26T13:35:20.824 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr: peer_sites: 2023-08-26T13:35:20.824 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr: name: cluster2 2023-08-26T13:35:20.824 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr: state: up+stopped 2023-08-26T13:35:20.824 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr: description: local image is primary 2023-08-26T13:35:20.825 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr: last_update: 2023-08-26 13:35:15 2023-08-26T13:35:20.898 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ grep '^ state: .*up+replaying' /tmp/tmp.TfDn6BcQ3N/cluster1-mirror-test.mirror_status 2023-08-26T13:35:20.899 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ return 1 2023-08-26T13:35:20.899 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-08-26T13:35:20.899 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ sleep 8 2023-08-26T13:35:25.451 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.0 with skip (0.45513275150596466) > weight (0.25) 2023-08-26T13:35:25.452 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.1 with skip (0.8731774343353816) > weight (0.25) 2023-08-26T13:35:25.452 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.2 with skip (0.8194463842005173) > weight (0.25) 2023-08-26T13:35:25.452 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.3 with skip (0.5996453617913636) > weight (0.25) 2023-08-26T13:35:25.452 INFO:tasks.rbd_mirror_thrash:waiting for 106 secs before thrashing 2023-08-26T13:35:28.901 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ test_status_in_pool_dir cluster1 mirror test up+replaying '' '' 2023-08-26T13:35:28.901 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster1 2023-08-26T13:35:28.902 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T13:35:28.902 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=test 2023-08-26T13:35:28.902 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local state_pattern=up+replaying 2023-08-26T13:35:28.902 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local description_pattern= 2023-08-26T13:35:28.903 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local service_pattern= 2023-08-26T13:35:28.903 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ mkfname cluster1-mirror-test.mirror_status 2023-08-26T13:35:28.903 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ echo cluster1-mirror-test.mirror_status 2023-08-26T13:35:28.903 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ sed -e 's|[/ ]|_|g' 2023-08-26T13:35:28.904 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local status_log=/tmp/tmp.TfDn6BcQ3N/cluster1-mirror-test.mirror_status 2023-08-26T13:35:28.904 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ CEPH_ARGS= 2023-08-26T13:35:28.904 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rbd --cluster cluster1 mirror image status mirror/test 2023-08-26T13:35:28.904 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ tee /tmp/tmp.TfDn6BcQ3N/cluster1-mirror-test.mirror_status 2023-08-26T13:35:28.950 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:test: 2023-08-26T13:35:28.951 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr: global_id: e2f2554d-659e-4467-b7f8-dddd7ccb0ee9 2023-08-26T13:35:28.951 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr: state: down+unknown 2023-08-26T13:35:28.951 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr: description: status not found 2023-08-26T13:35:28.951 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr: last_update: 2023-08-26T13:35:28.952 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr: peer_sites: 2023-08-26T13:35:28.952 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr: name: cluster2 2023-08-26T13:35:28.952 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr: state: up+stopped 2023-08-26T13:35:28.952 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr: description: local image is primary 2023-08-26T13:35:28.952 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr: last_update: 2023-08-26 13:35:15 2023-08-26T13:35:28.955 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ grep '^ state: .*up+replaying' /tmp/tmp.TfDn6BcQ3N/cluster1-mirror-test.mirror_status 2023-08-26T13:35:28.956 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ return 1 2023-08-26T13:35:28.957 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-08-26T13:35:28.957 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ sleep 8 2023-08-26T13:35:36.958 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ test_status_in_pool_dir cluster1 mirror test up+replaying '' '' 2023-08-26T13:35:36.959 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster1 2023-08-26T13:35:36.959 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T13:35:36.959 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=test 2023-08-26T13:35:36.959 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local state_pattern=up+replaying 2023-08-26T13:35:36.960 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local description_pattern= 2023-08-26T13:35:36.960 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local service_pattern= 2023-08-26T13:35:36.960 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ mkfname cluster1-mirror-test.mirror_status 2023-08-26T13:35:36.960 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ echo cluster1-mirror-test.mirror_status 2023-08-26T13:35:36.960 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ sed -e 's|[/ ]|_|g' 2023-08-26T13:35:36.961 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local status_log=/tmp/tmp.TfDn6BcQ3N/cluster1-mirror-test.mirror_status 2023-08-26T13:35:36.961 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ CEPH_ARGS= 2023-08-26T13:35:36.962 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rbd --cluster cluster1 mirror image status mirror/test 2023-08-26T13:35:36.962 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ tee /tmp/tmp.TfDn6BcQ3N/cluster1-mirror-test.mirror_status 2023-08-26T13:35:37.008 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:test: 2023-08-26T13:35:37.008 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr: global_id: e2f2554d-659e-4467-b7f8-dddd7ccb0ee9 2023-08-26T13:35:37.008 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr: state: down+unknown 2023-08-26T13:35:37.008 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr: description: status not found 2023-08-26T13:35:37.008 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr: last_update: 2023-08-26T13:35:37.009 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr: peer_sites: 2023-08-26T13:35:37.009 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr: name: cluster2 2023-08-26T13:35:37.009 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr: state: up+stopped 2023-08-26T13:35:37.009 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr: description: local image is primary 2023-08-26T13:35:37.009 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr: last_update: 2023-08-26 13:35:15 2023-08-26T13:35:37.012 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ grep '^ state: .*up+replaying' /tmp/tmp.TfDn6BcQ3N/cluster1-mirror-test.mirror_status 2023-08-26T13:35:37.014 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ return 1 2023-08-26T13:35:37.014 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-08-26T13:35:37.014 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ sleep 8 2023-08-26T13:35:45.016 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ test_status_in_pool_dir cluster1 mirror test up+replaying '' '' 2023-08-26T13:35:45.016 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster1 2023-08-26T13:35:45.016 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T13:35:45.017 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=test 2023-08-26T13:35:45.017 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local state_pattern=up+replaying 2023-08-26T13:35:45.017 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local description_pattern= 2023-08-26T13:35:45.017 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local service_pattern= 2023-08-26T13:35:45.017 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ mkfname cluster1-mirror-test.mirror_status 2023-08-26T13:35:45.018 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ echo cluster1-mirror-test.mirror_status 2023-08-26T13:35:45.018 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ sed -e 's|[/ ]|_|g' 2023-08-26T13:35:45.019 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local status_log=/tmp/tmp.TfDn6BcQ3N/cluster1-mirror-test.mirror_status 2023-08-26T13:35:45.019 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ CEPH_ARGS= 2023-08-26T13:35:45.019 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rbd --cluster cluster1 mirror image status mirror/test 2023-08-26T13:35:45.019 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ tee /tmp/tmp.TfDn6BcQ3N/cluster1-mirror-test.mirror_status 2023-08-26T13:35:45.071 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:test: 2023-08-26T13:35:45.071 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr: global_id: e2f2554d-659e-4467-b7f8-dddd7ccb0ee9 2023-08-26T13:35:45.071 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr: state: up+replaying 2023-08-26T13:35:45.071 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr: description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":0.0,"last_snapshot_bytes":0,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1693056888,"remote_snapshot_timestamp":1693056888,"replay_state":"idle"} 2023-08-26T13:35:45.071 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr: service: mirror.2 on smithi124 2023-08-26T13:35:45.072 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr: last_update: 2023-08-26 13:35:39 2023-08-26T13:35:45.072 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr: peer_sites: 2023-08-26T13:35:45.072 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr: name: cluster2 2023-08-26T13:35:45.072 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr: state: up+stopped 2023-08-26T13:35:45.072 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr: description: local image is primary 2023-08-26T13:35:45.073 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr: last_update: 2023-08-26 13:35:15 2023-08-26T13:35:45.075 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ grep '^ state: .*up+replaying' /tmp/tmp.TfDn6BcQ3N/cluster1-mirror-test.mirror_status 2023-08-26T13:35:45.076 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: state: up+replaying 2023-08-26T13:35:45.077 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ grep '^ description: .*' /tmp/tmp.TfDn6BcQ3N/cluster1-mirror-test.mirror_status 2023-08-26T13:35:45.077 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":0.0,"last_snapshot_bytes":0,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1693056888,"remote_snapshot_timestamp":1693056888,"replay_state":"idle"} 2023-08-26T13:35:45.078 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ '[' -n '' ']' 2023-08-26T13:35:45.078 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ echo up+replaying 2023-08-26T13:35:45.078 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ grep '^up+' 2023-08-26T13:35:45.079 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:up+replaying 2023-08-26T13:35:45.079 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ grep 'service: *mirror..* on ' /tmp/tmp.TfDn6BcQ3N/cluster1-mirror-test.mirror_status 2023-08-26T13:35:45.080 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: service: mirror.2 on smithi124 2023-08-26T13:35:45.080 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' /tmp/tmp.TfDn6BcQ3N/cluster1-mirror-test.mirror_status 2023-08-26T13:35:45.082 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local 'last_update=2023-08-26 13:35:39' 2023-08-26T13:35:45.082 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ test_mirror_pool_status_verbose cluster1 mirror test up+replaying '2023-08-26 13:35:39' 2023-08-26T13:35:45.082 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster1 2023-08-26T13:35:45.082 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T13:35:45.082 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=test 2023-08-26T13:35:45.083 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local state_pattern=up+replaying 2023-08-26T13:35:45.083 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local 'prev_last_update=2023-08-26 13:35:39' 2023-08-26T13:35:45.083 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ mkfname cluster1-mirror.mirror_status 2023-08-26T13:35:45.083 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ echo cluster1-mirror.mirror_status 2023-08-26T13:35:45.083 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ sed -e 's|[/ ]|_|g' 2023-08-26T13:35:45.084 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local status_log=/tmp/tmp.TfDn6BcQ3N/cluster1-mirror.mirror_status 2023-08-26T13:35:45.084 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2023-08-26T13:35:45.120 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:rbd: failed to query services: (13) Permission denied 2023-08-26T13:35:45.144 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local last_update state 2023-08-26T13:35:45.144 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update' 2023-08-26T13:35:45.145 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ last_update='2023-08-26 13:35:39' 2023-08-26T13:35:45.146 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state' 2023-08-26T13:35:45.148 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ state=up+replaying 2023-08-26T13:35:45.148 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ echo up+replaying 2023-08-26T13:35:45.148 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ grep up+replaying 2023-08-26T13:35:45.148 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:up+replaying 2023-08-26T13:35:45.149 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ return 0 2023-08-26T13:35:45.150 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ return 0 2023-08-26T13:35:45.150 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ snap_name=snap3 2023-08-26T13:35:45.150 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ create_snap cluster2 mirror test snap3 2023-08-26T13:35:45.150 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster2 2023-08-26T13:35:45.150 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T13:35:45.150 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=test 2023-08-26T13:35:45.151 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local snap_name=snap3 2023-08-26T13:35:45.151 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rbd --cluster cluster2 -p mirror snap create test@snap3 --debug-rbd=20 --debug-journaler=20 2023-08-26T13:35:46.614 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ wait_for_image_replay_started cluster1 mirror test 2023-08-26T13:35:46.614 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster1 2023-08-26T13:35:46.614 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T13:35:46.614 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=test 2023-08-26T13:35:46.615 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ wait_for_image_replay_state cluster1 mirror test started 2023-08-26T13:35:46.615 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster1 2023-08-26T13:35:46.615 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T13:35:46.615 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=test 2023-08-26T13:35:46.615 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local state=started 2023-08-26T13:35:46.616 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local s 2023-08-26T13:35:46.616 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-08-26T13:35:46.616 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ sleep 1 2023-08-26T13:35:47.617 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ test_image_replay_state cluster1 mirror test started 2023-08-26T13:35:47.618 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster1 2023-08-26T13:35:47.618 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T13:35:47.618 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=test 2023-08-26T13:35:47.619 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local test_state=started 2023-08-26T13:35:47.619 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local status_result 2023-08-26T13:35:47.620 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local current_state=stopped 2023-08-26T13:35:47.620 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ admin_daemons cluster1 rbd mirror status mirror/test 2023-08-26T13:35:47.620 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ grep -i state 2023-08-26T13:35:47.621 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local cluster_instance=cluster1 2023-08-26T13:35:47.621 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ shift 2023-08-26T13:35:47.621 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local cluster=cluster1 2023-08-26T13:35:47.622 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local instance=cluster1 2023-08-26T13:35:47.622 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local loop_instance 2023-08-26T13:35:47.622 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-08-26T13:35:47.623 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ sleep 0 2023-08-26T13:35:47.623 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ '[' cluster1 '!=' cluster1 ']' 2023-08-26T13:35:47.623 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ seq 0 3 2023-08-26T13:35:47.629 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-08-26T13:35:47.630 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test 2023-08-26T13:35:47.630 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local cluster=cluster1:0 2023-08-26T13:35:47.630 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ shift 2023-08-26T13:35:47.631 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local instance 2023-08-26T13:35:47.631 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ set_cluster_instance cluster1:0 cluster instance 2023-08-26T13:35:47.631 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local val=cluster1:0 2023-08-26T13:35:47.632 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local cluster_var_name=cluster 2023-08-26T13:35:47.632 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local instance_var_name=instance 2023-08-26T13:35:47.632 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ cluster=cluster1 2023-08-26T13:35:47.633 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ instance=0 2023-08-26T13:35:47.633 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ '[' 0 = cluster1:0 ']' 2023-08-26T13:35:47.634 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ eval cluster=cluster1 2023-08-26T13:35:47.634 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ cluster=cluster1 2023-08-26T13:35:47.634 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ eval instance=0 2023-08-26T13:35:47.635 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ instance=0 2023-08-26T13:35:47.635 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ daemon_asok_file cluster1:0 cluster1 2023-08-26T13:35:47.635 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ local local_cluster=cluster1:0 2023-08-26T13:35:47.636 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ local cluster=cluster1 2023-08-26T13:35:47.636 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ local instance 2023-08-26T13:35:47.636 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2023-08-26T13:35:47.637 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ local val=cluster1:0 2023-08-26T13:35:47.637 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ local cluster_var_name=local_cluster 2023-08-26T13:35:47.638 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ local instance_var_name=instance 2023-08-26T13:35:47.638 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ cluster=cluster1 2023-08-26T13:35:47.638 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ instance=0 2023-08-26T13:35:47.639 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ '[' 0 = cluster1:0 ']' 2023-08-26T13:35:47.639 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ eval local_cluster=cluster1 2023-08-26T13:35:47.639 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++++ local_cluster=cluster1 2023-08-26T13:35:47.640 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ eval instance=0 2023-08-26T13:35:47.640 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++++ instance=0 2023-08-26T13:35:47.641 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2023-08-26T13:35:47.642 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-08-26T13:35:47.642 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-08-26T13:35:47.643 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-08-26T13:35:47.643 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2023-08-26T13:35:47.746 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:Invalid command: unused arguments: ['mirror/test'] 2023-08-26T13:35:47.747 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:rbd mirror status : get status for rbd mirror 2023-08-26T13:35:47.747 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:admin_socket: invalid command 2023-08-26T13:35:47.750 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-08-26T13:35:47.750 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/test 2023-08-26T13:35:47.750 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local cluster=cluster1:1 2023-08-26T13:35:47.751 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ shift 2023-08-26T13:35:47.751 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local instance 2023-08-26T13:35:47.751 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ set_cluster_instance cluster1:1 cluster instance 2023-08-26T13:35:47.751 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local val=cluster1:1 2023-08-26T13:35:47.752 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local cluster_var_name=cluster 2023-08-26T13:35:47.752 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local instance_var_name=instance 2023-08-26T13:35:47.752 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ cluster=cluster1 2023-08-26T13:35:47.752 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ instance=1 2023-08-26T13:35:47.752 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ '[' 1 = cluster1:1 ']' 2023-08-26T13:35:47.753 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ eval cluster=cluster1 2023-08-26T13:35:47.753 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ cluster=cluster1 2023-08-26T13:35:47.753 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ eval instance=1 2023-08-26T13:35:47.753 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ instance=1 2023-08-26T13:35:47.754 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ daemon_asok_file cluster1:1 cluster1 2023-08-26T13:35:47.754 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ local local_cluster=cluster1:1 2023-08-26T13:35:47.754 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ local cluster=cluster1 2023-08-26T13:35:47.754 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ local instance 2023-08-26T13:35:47.754 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2023-08-26T13:35:47.755 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ local val=cluster1:1 2023-08-26T13:35:47.755 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ local cluster_var_name=local_cluster 2023-08-26T13:35:47.755 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ local instance_var_name=instance 2023-08-26T13:35:47.755 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ cluster=cluster1 2023-08-26T13:35:47.755 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ instance=1 2023-08-26T13:35:47.756 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ '[' 1 = cluster1:1 ']' 2023-08-26T13:35:47.756 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ eval local_cluster=cluster1 2023-08-26T13:35:47.756 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++++ local_cluster=cluster1 2023-08-26T13:35:47.756 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ eval instance=1 2023-08-26T13:35:47.757 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++++ instance=1 2023-08-26T13:35:47.757 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2023-08-26T13:35:47.769 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-08-26T13:35:47.769 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-08-26T13:35:47.769 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-08-26T13:35:47.769 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/test 2023-08-26T13:35:47.863 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:Invalid command: unused arguments: ['mirror/test'] 2023-08-26T13:35:47.863 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:rbd mirror status : get status for rbd mirror 2023-08-26T13:35:47.864 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:admin_socket: invalid command 2023-08-26T13:35:47.866 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-08-26T13:35:47.867 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/test 2023-08-26T13:35:47.867 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local cluster=cluster1:2 2023-08-26T13:35:47.867 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ shift 2023-08-26T13:35:47.867 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local instance 2023-08-26T13:35:47.867 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ set_cluster_instance cluster1:2 cluster instance 2023-08-26T13:35:47.868 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local val=cluster1:2 2023-08-26T13:35:47.868 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local cluster_var_name=cluster 2023-08-26T13:35:47.868 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local instance_var_name=instance 2023-08-26T13:35:47.868 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ cluster=cluster1 2023-08-26T13:35:47.868 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ instance=2 2023-08-26T13:35:47.869 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ '[' 2 = cluster1:2 ']' 2023-08-26T13:35:47.869 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ eval cluster=cluster1 2023-08-26T13:35:47.869 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ cluster=cluster1 2023-08-26T13:35:47.869 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ eval instance=2 2023-08-26T13:35:47.869 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ instance=2 2023-08-26T13:35:47.869 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ daemon_asok_file cluster1:2 cluster1 2023-08-26T13:35:47.870 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ local local_cluster=cluster1:2 2023-08-26T13:35:47.870 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ local cluster=cluster1 2023-08-26T13:35:47.870 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ local instance 2023-08-26T13:35:47.870 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2023-08-26T13:35:47.870 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ local val=cluster1:2 2023-08-26T13:35:47.871 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ local cluster_var_name=local_cluster 2023-08-26T13:35:47.871 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ local instance_var_name=instance 2023-08-26T13:35:47.871 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ cluster=cluster1 2023-08-26T13:35:47.871 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ instance=2 2023-08-26T13:35:47.871 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ '[' 2 = cluster1:2 ']' 2023-08-26T13:35:47.872 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ eval local_cluster=cluster1 2023-08-26T13:35:47.872 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++++ local_cluster=cluster1 2023-08-26T13:35:47.872 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ eval instance=2 2023-08-26T13:35:47.872 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++++ instance=2 2023-08-26T13:35:47.872 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2023-08-26T13:35:47.885 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-08-26T13:35:47.885 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-08-26T13:35:47.885 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-08-26T13:35:47.885 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/test 2023-08-26T13:35:47.988 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ return 0 2023-08-26T13:35:47.990 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ status_result=' "state": "Replaying"' 2023-08-26T13:35:47.991 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ echo ' "state": "Replaying"' 2023-08-26T13:35:47.991 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ grep -i Replaying 2023-08-26T13:35:47.991 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: "state": "Replaying" 2023-08-26T13:35:47.994 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ current_state=started 2023-08-26T13:35:47.994 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ test started = started 2023-08-26T13:35:47.994 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ return 0 2023-08-26T13:35:47.994 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror test 2023-08-26T13:35:47.995 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local local_cluster=cluster1 2023-08-26T13:35:47.995 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster2 2023-08-26T13:35:47.995 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T13:35:47.995 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=test 2023-08-26T13:35:47.995 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ '[' snapshot = journal ']' 2023-08-26T13:35:47.996 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ '[' snapshot = snapshot ']' 2023-08-26T13:35:47.996 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror test 2023-08-26T13:35:47.996 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local local_cluster=cluster1 2023-08-26T13:35:47.996 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster2 2023-08-26T13:35:47.996 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T13:35:47.997 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=test 2023-08-26T13:35:47.997 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ mkfname cluster2-mirror-test.status 2023-08-26T13:35:47.997 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ echo cluster2-mirror-test.status 2023-08-26T13:35:47.997 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ sed -e 's|[/ ]|_|g' 2023-08-26T13:35:47.997 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local status_log=/tmp/tmp.TfDn6BcQ3N/cluster2-mirror-test.status 2023-08-26T13:35:47.998 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ mkfname cluster1-mirror-test.status 2023-08-26T13:35:47.998 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ echo cluster1-mirror-test.status 2023-08-26T13:35:47.998 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ sed -e 's|[/ ]|_|g' 2023-08-26T13:35:47.998 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local local_status_log=/tmp/tmp.TfDn6BcQ3N/cluster1-mirror-test.status 2023-08-26T13:35:47.998 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ mirror_image_snapshot cluster2 mirror test 2023-08-26T13:35:47.998 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster2 2023-08-26T13:35:47.999 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T13:35:47.999 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=test 2023-08-26T13:35:47.999 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/test 2023-08-26T13:35:49.625 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:Snapshot ID: 13 2023-08-26T13:35:49.644 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ get_newest_mirror_snapshot cluster2 mirror test /tmp/tmp.TfDn6BcQ3N/cluster2-mirror-test.status 2023-08-26T13:35:49.645 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster2 2023-08-26T13:35:49.645 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T13:35:49.645 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=test 2023-08-26T13:35:49.645 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local log=/tmp/tmp.TfDn6BcQ3N/cluster2-mirror-test.status 2023-08-26T13:35:49.645 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rbd --cluster cluster2 snap list --all mirror/test --format xml 2023-08-26T13:35:49.646 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-08-26T13:35:49.712 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ xmlstarlet sel -t -v //snapshot/id 2023-08-26T13:35:49.714 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local snapshot_id=13 2023-08-26T13:35:49.714 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ true 2023-08-26T13:35:49.714 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-08-26T13:35:49.714 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ sleep 0.2 2023-08-26T13:35:49.915 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.TfDn6BcQ3N/cluster1-mirror-test.status 2023-08-26T13:35:49.916 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster1 2023-08-26T13:35:49.916 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T13:35:49.916 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=test 2023-08-26T13:35:49.916 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local log=/tmp/tmp.TfDn6BcQ3N/cluster1-mirror-test.status 2023-08-26T13:35:49.916 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2023-08-26T13:35:49.917 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-08-26T13:35:49.964 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ true 2023-08-26T13:35:49.964 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-08-26T13:35:49.965 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:-:1.1: Document is empty 2023-08-26T13:35:49.965 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr: 2023-08-26T13:35:49.966 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:^ 2023-08-26T13:35:49.967 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local primary_snapshot_id= 2023-08-26T13:35:49.967 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ test 13 = '' 2023-08-26T13:35:49.967 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-08-26T13:35:49.967 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ sleep 0.4 2023-08-26T13:35:50.367 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.TfDn6BcQ3N/cluster1-mirror-test.status 2023-08-26T13:35:50.367 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster1 2023-08-26T13:35:50.367 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T13:35:50.368 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=test 2023-08-26T13:35:50.368 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local log=/tmp/tmp.TfDn6BcQ3N/cluster1-mirror-test.status 2023-08-26T13:35:50.368 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2023-08-26T13:35:50.368 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-08-26T13:35:50.415 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ true 2023-08-26T13:35:50.415 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-08-26T13:35:50.416 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:-:1.1: Document is empty 2023-08-26T13:35:50.416 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr: 2023-08-26T13:35:50.416 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:^ 2023-08-26T13:35:50.417 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local primary_snapshot_id= 2023-08-26T13:35:50.417 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ test 13 = '' 2023-08-26T13:35:50.417 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-08-26T13:35:50.418 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ sleep 0.8 2023-08-26T13:35:51.218 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.TfDn6BcQ3N/cluster1-mirror-test.status 2023-08-26T13:35:51.218 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster1 2023-08-26T13:35:51.219 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T13:35:51.219 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=test 2023-08-26T13:35:51.219 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local log=/tmp/tmp.TfDn6BcQ3N/cluster1-mirror-test.status 2023-08-26T13:35:51.219 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2023-08-26T13:35:51.219 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-08-26T13:35:51.304 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ true 2023-08-26T13:35:51.304 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-08-26T13:35:51.306 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:-:1.1: Document is empty 2023-08-26T13:35:51.306 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr: 2023-08-26T13:35:51.306 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:^ 2023-08-26T13:35:51.307 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local primary_snapshot_id= 2023-08-26T13:35:51.307 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ test 13 = '' 2023-08-26T13:35:51.307 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-08-26T13:35:51.307 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ sleep 1.6 2023-08-26T13:35:52.908 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.TfDn6BcQ3N/cluster1-mirror-test.status 2023-08-26T13:35:52.909 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster1 2023-08-26T13:35:52.909 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T13:35:52.909 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=test 2023-08-26T13:35:52.909 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local log=/tmp/tmp.TfDn6BcQ3N/cluster1-mirror-test.status 2023-08-26T13:35:52.909 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2023-08-26T13:35:52.910 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-08-26T13:35:52.960 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-08-26T13:35:52.962 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local primary_snapshot_id=13 2023-08-26T13:35:52.962 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ test 13 = 13 2023-08-26T13:35:52.963 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ return 0 2023-08-26T13:35:52.963 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ wait_for_snap_present cluster1 mirror test snap3 2023-08-26T13:35:52.963 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster1 2023-08-26T13:35:52.963 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T13:35:52.964 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=test 2023-08-26T13:35:52.964 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local snap_name=snap3 2023-08-26T13:35:52.964 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local s 2023-08-26T13:35:52.964 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2023-08-26T13:35:52.964 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ sleep 1 2023-08-26T13:35:53.963 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rbd --cluster cluster1 info mirror/test@snap3 2023-08-26T13:35:54.008 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:rbd image 'test': 2023-08-26T13:35:54.009 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: size 512 MiB in 128 objects 2023-08-26T13:35:54.009 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: order 22 (4 MiB objects) 2023-08-26T13:35:54.009 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: snapshot_count: 6 2023-08-26T13:35:54.009 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: id: 10ab632cdafc 2023-08-26T13:35:54.009 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: block_name_prefix: rbd_data.10ab632cdafc 2023-08-26T13:35:54.010 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: format: 2 2023-08-26T13:35:54.010 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: features: layering, exclusive-lock, non-primary 2023-08-26T13:35:54.010 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: op_features: 2023-08-26T13:35:54.010 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: flags: 2023-08-26T13:35:54.011 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: create_timestamp: Sat Aug 26 13:33:17 2023 2023-08-26T13:35:54.011 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: access_timestamp: Sat Aug 26 13:34:54 2023 2023-08-26T13:35:54.011 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: modify_timestamp: Sat Aug 26 13:33:17 2023 2023-08-26T13:35:54.011 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: protected: False 2023-08-26T13:35:54.011 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: mirroring state: enabled 2023-08-26T13:35:54.012 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: mirroring mode: snapshot 2023-08-26T13:35:54.012 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: mirroring global id: e2f2554d-659e-4467-b7f8-dddd7ccb0ee9 2023-08-26T13:35:54.012 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: mirroring primary: false 2023-08-26T13:35:54.013 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ return 0 2023-08-26T13:35:54.013 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ '[' -n snap2-clean ']' 2023-08-26T13:35:54.014 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ compare_image_snaps mirror test snap2-clean 2023-08-26T13:35:54.014 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T13:35:54.014 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=test 2023-08-26T13:35:54.014 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local snap_name=snap2-clean 2023-08-26T13:35:54.014 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local ret=0 2023-08-26T13:35:54.015 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local rmt_export=/tmp/tmp.TfDn6BcQ3N/cluster2-mirror-test.export 2023-08-26T13:35:54.015 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local loc_export=/tmp/tmp.TfDn6BcQ3N/cluster1-mirror-test.export 2023-08-26T13:35:54.015 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rm -f /tmp/tmp.TfDn6BcQ3N/cluster2-mirror-test.export /tmp/tmp.TfDn6BcQ3N/cluster1-mirror-test.export 2023-08-26T13:35:54.015 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rbd --cluster cluster2 -p mirror export test@snap2-clean /tmp/tmp.TfDn6BcQ3N/cluster2-mirror-test.export 2023-08-26T13:35:54.581 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2023-08-26T13:35:54.592 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rbd --cluster cluster1 -p mirror export test@snap2-clean /tmp/tmp.TfDn6BcQ3N/cluster1-mirror-test.export 2023-08-26T13:35:55.238 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2023-08-26T13:35:55.246 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ cmp /tmp/tmp.TfDn6BcQ3N/cluster2-mirror-test.export /tmp/tmp.TfDn6BcQ3N/cluster1-mirror-test.export 2023-08-26T13:35:55.459 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rm -f /tmp/tmp.TfDn6BcQ3N/cluster2-mirror-test.export /tmp/tmp.TfDn6BcQ3N/cluster1-mirror-test.export 2023-08-26T13:35:55.606 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ return 0 2023-08-26T13:35:55.607 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ compare_image_snaps mirror test snap3 2023-08-26T13:35:55.607 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T13:35:55.607 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=test 2023-08-26T13:35:55.607 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local snap_name=snap3 2023-08-26T13:35:55.608 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local ret=0 2023-08-26T13:35:55.608 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local rmt_export=/tmp/tmp.TfDn6BcQ3N/cluster2-mirror-test.export 2023-08-26T13:35:55.608 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local loc_export=/tmp/tmp.TfDn6BcQ3N/cluster1-mirror-test.export 2023-08-26T13:35:55.608 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rm -f /tmp/tmp.TfDn6BcQ3N/cluster2-mirror-test.export /tmp/tmp.TfDn6BcQ3N/cluster1-mirror-test.export 2023-08-26T13:35:55.609 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rbd --cluster cluster2 -p mirror export test@snap3 /tmp/tmp.TfDn6BcQ3N/cluster2-mirror-test.export 2023-08-26T13:35:56.186 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2023-08-26T13:35:56.197 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rbd --cluster cluster1 -p mirror export test@snap3 /tmp/tmp.TfDn6BcQ3N/cluster1-mirror-test.export 2023-08-26T13:35:56.952 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2023-08-26T13:35:56.962 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ cmp /tmp/tmp.TfDn6BcQ3N/cluster2-mirror-test.export /tmp/tmp.TfDn6BcQ3N/cluster1-mirror-test.export 2023-08-26T13:35:57.172 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rm -f /tmp/tmp.TfDn6BcQ3N/cluster2-mirror-test.export /tmp/tmp.TfDn6BcQ3N/cluster1-mirror-test.export 2023-08-26T13:35:57.315 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ return 0 2023-08-26T13:35:57.315 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ clean_snap_name=snap3-clean 2023-08-26T13:35:57.315 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ create_snap cluster2 mirror test snap3-clean 2023-08-26T13:35:57.315 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster2 2023-08-26T13:35:57.316 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T13:35:57.316 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=test 2023-08-26T13:35:57.316 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local snap_name=snap3-clean 2023-08-26T13:35:57.316 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rbd --cluster cluster2 -p mirror snap create test@snap3-clean --debug-rbd=20 --debug-journaler=20 2023-08-26T13:35:57.574 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ for i in `seq 1 10` 2023-08-26T13:35:57.574 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ stress_write_image cluster2 mirror test 2023-08-26T13:35:57.575 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster2 2023-08-26T13:35:57.575 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T13:35:57.575 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=test 2023-08-26T13:35:57.575 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2023-08-26T13:35:57.576 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local duration=9 2023-08-26T13:35:57.577 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ set +e 2023-08-26T13:35:57.578 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ timeout 9s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror test --debug-rbd=20 --debug-journaler=20 2023-08-26T13:35:57.736 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:100 IOs, 6289408 bytes 2023-08-26T13:35:57.916 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:200 IOs, 51822592 bytes 2023-08-26T13:35:58.095 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:300 IOs, 106940416 bytes 2023-08-26T13:35:58.307 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:400 IOs, 178630656 bytes 2023-08-26T13:35:58.407 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:500 IOs, 179226624 bytes 2023-08-26T13:35:58.506 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:600 IOs, 198795264 bytes 2023-08-26T13:35:58.592 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:700 IOs, 199386112 bytes 2023-08-26T13:35:58.657 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:800 IOs, 203277312 bytes 2023-08-26T13:35:58.772 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:900 IOs, 228057088 bytes 2023-08-26T13:35:58.915 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:1000 IOs, 262991872 bytes 2023-08-26T13:35:58.984 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:1100 IOs, 263592960 bytes 2023-08-26T13:35:59.093 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:1200 IOs, 287276032 bytes 2023-08-26T13:35:59.322 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:1300 IOs, 371122176 bytes 2023-08-26T13:35:59.420 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:1400 IOs, 385104896 bytes 2023-08-26T13:35:59.549 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:1500 IOs, 430240768 bytes 2023-08-26T13:35:59.701 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:1600 IOs, 462595072 bytes 2023-08-26T13:35:59.790 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:1700 IOs, 481994752 bytes 2023-08-26T13:35:59.892 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:1800 IOs, 495815680 bytes 2023-08-26T13:36:00.032 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:1900 IOs, 531226624 bytes 2023-08-26T13:36:00.126 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:2000 IOs, 541248512 bytes 2023-08-26T13:36:00.232 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:2100 IOs, 569842688 bytes 2023-08-26T13:36:00.383 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:2200 IOs, 606516224 bytes 2023-08-26T13:36:00.498 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:2300 IOs, 625558528 bytes 2023-08-26T13:36:00.654 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:2400 IOs, 666464256 bytes 2023-08-26T13:36:00.831 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:2500 IOs, 722475008 bytes 2023-08-26T13:36:00.896 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:2600 IOs, 723104768 bytes 2023-08-26T13:36:01.115 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:2700 IOs, 806162432 bytes 2023-08-26T13:36:01.223 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:2800 IOs, 828510208 bytes 2023-08-26T13:36:01.421 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:2900 IOs, 895648768 bytes 2023-08-26T13:36:01.562 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:3000 IOs, 944619520 bytes 2023-08-26T13:36:01.764 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:3100 IOs, 1002902528 bytes 2023-08-26T13:36:01.924 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:3200 IOs, 1055569920 bytes 2023-08-26T13:36:02.086 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:3300 IOs, 1095106560 bytes 2023-08-26T13:36:02.220 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:3400 IOs, 1130658816 bytes 2023-08-26T13:36:02.431 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:3500 IOs, 1196600320 bytes 2023-08-26T13:36:02.555 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:3600 IOs, 1224181760 bytes 2023-08-26T13:36:02.707 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:3700 IOs, 1282864128 bytes 2023-08-26T13:36:02.880 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:3800 IOs, 1324234752 bytes 2023-08-26T13:36:03.087 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:3900 IOs, 1378850816 bytes 2023-08-26T13:36:03.245 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:4000 IOs, 1421494272 bytes 2023-08-26T13:36:03.356 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:4100 IOs, 1439226880 bytes 2023-08-26T13:36:03.528 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:4200 IOs, 1489998848 bytes 2023-08-26T13:36:03.652 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:4300 IOs, 1522217984 bytes 2023-08-26T13:36:03.826 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:4400 IOs, 1601699840 bytes 2023-08-26T13:36:03.999 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:4500 IOs, 1631212544 bytes 2023-08-26T13:36:04.182 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:4600 IOs, 1687678976 bytes 2023-08-26T13:36:04.341 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:4700 IOs, 1737357312 bytes 2023-08-26T13:36:04.476 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:4800 IOs, 1767748608 bytes 2023-08-26T13:36:04.584 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:4900 IOs, 1787658240 bytes 2023-08-26T13:36:04.687 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:5000 IOs, 1806970880 bytes 2023-08-26T13:36:04.851 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:5100 IOs, 1855653888 bytes 2023-08-26T13:36:05.002 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:5200 IOs, 1895761920 bytes 2023-08-26T13:36:05.137 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:5300 IOs, 1927029760 bytes 2023-08-26T13:36:05.324 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:5400 IOs, 1999923200 bytes 2023-08-26T13:36:05.490 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:5500 IOs, 2032173056 bytes 2023-08-26T13:36:05.595 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:5600 IOs, 2051641344 bytes 2023-08-26T13:36:05.702 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:5700 IOs, 2062309376 bytes 2023-08-26T13:36:05.857 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:5800 IOs, 2105379840 bytes 2023-08-26T13:36:05.993 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:5900 IOs, 2146264064 bytes 2023-08-26T13:36:06.216 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:6000 IOs, 2231846912 bytes 2023-08-26T13:36:06.421 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:6100 IOs, 2308496384 bytes 2023-08-26T13:36:06.526 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:6200 IOs, 2333545472 bytes 2023-08-26T13:36:06.580 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ error_code=124 2023-08-26T13:36:06.580 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ set -e 2023-08-26T13:36:06.580 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ '[' 124 -eq 124 ']' 2023-08-26T13:36:06.580 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ return 0 2023-08-26T13:36:06.581 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ wait_for_status_in_pool_dir cluster1 mirror test up+replaying 2023-08-26T13:36:06.581 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster1 2023-08-26T13:36:06.581 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T13:36:06.581 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=test 2023-08-26T13:36:06.581 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local state_pattern=up+replaying 2023-08-26T13:36:06.582 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local description_pattern= 2023-08-26T13:36:06.582 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local service_pattern= 2023-08-26T13:36:06.582 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-08-26T13:36:06.582 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ sleep 1 2023-08-26T13:36:07.581 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ test_status_in_pool_dir cluster1 mirror test up+replaying '' '' 2023-08-26T13:36:07.581 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster1 2023-08-26T13:36:07.581 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T13:36:07.582 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=test 2023-08-26T13:36:07.582 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local state_pattern=up+replaying 2023-08-26T13:36:07.582 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local description_pattern= 2023-08-26T13:36:07.582 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local service_pattern= 2023-08-26T13:36:07.582 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ mkfname cluster1-mirror-test.mirror_status 2023-08-26T13:36:07.583 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ echo cluster1-mirror-test.mirror_status 2023-08-26T13:36:07.583 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ sed -e 's|[/ ]|_|g' 2023-08-26T13:36:07.583 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local status_log=/tmp/tmp.TfDn6BcQ3N/cluster1-mirror-test.mirror_status 2023-08-26T13:36:07.584 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ CEPH_ARGS= 2023-08-26T13:36:07.584 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rbd --cluster cluster1 mirror image status mirror/test 2023-08-26T13:36:07.584 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ tee /tmp/tmp.TfDn6BcQ3N/cluster1-mirror-test.mirror_status 2023-08-26T13:36:07.635 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:test: 2023-08-26T13:36:07.636 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr: global_id: e2f2554d-659e-4467-b7f8-dddd7ccb0ee9 2023-08-26T13:36:07.636 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr: state: up+replaying 2023-08-26T13:36:07.636 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr: description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":0.0,"last_snapshot_bytes":0,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1693056888,"remote_snapshot_timestamp":1693056888,"replay_state":"idle"} 2023-08-26T13:36:07.636 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr: service: mirror.2 on smithi124 2023-08-26T13:36:07.637 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr: last_update: 2023-08-26 13:35:45 2023-08-26T13:36:07.637 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr: peer_sites: 2023-08-26T13:36:07.637 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr: name: cluster2 2023-08-26T13:36:07.637 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr: state: up+stopped 2023-08-26T13:36:07.638 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr: description: local image is primary 2023-08-26T13:36:07.638 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr: last_update: 2023-08-26 13:35:45 2023-08-26T13:36:07.640 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ grep '^ state: .*up+replaying' /tmp/tmp.TfDn6BcQ3N/cluster1-mirror-test.mirror_status 2023-08-26T13:36:07.641 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: state: up+replaying 2023-08-26T13:36:07.642 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ grep '^ description: .*' /tmp/tmp.TfDn6BcQ3N/cluster1-mirror-test.mirror_status 2023-08-26T13:36:07.642 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":0.0,"last_snapshot_bytes":0,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1693056888,"remote_snapshot_timestamp":1693056888,"replay_state":"idle"} 2023-08-26T13:36:07.643 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ '[' -n '' ']' 2023-08-26T13:36:07.643 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ echo up+replaying 2023-08-26T13:36:07.643 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ grep '^up+' 2023-08-26T13:36:07.644 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:up+replaying 2023-08-26T13:36:07.644 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ grep 'service: *mirror..* on ' /tmp/tmp.TfDn6BcQ3N/cluster1-mirror-test.mirror_status 2023-08-26T13:36:07.645 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: service: mirror.2 on smithi124 2023-08-26T13:36:07.645 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' /tmp/tmp.TfDn6BcQ3N/cluster1-mirror-test.mirror_status 2023-08-26T13:36:07.647 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local 'last_update=2023-08-26 13:35:45' 2023-08-26T13:36:07.647 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ test_mirror_pool_status_verbose cluster1 mirror test up+replaying '2023-08-26 13:35:45' 2023-08-26T13:36:07.647 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster1 2023-08-26T13:36:07.647 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T13:36:07.648 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=test 2023-08-26T13:36:07.648 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local state_pattern=up+replaying 2023-08-26T13:36:07.648 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local 'prev_last_update=2023-08-26 13:35:45' 2023-08-26T13:36:07.648 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ mkfname cluster1-mirror.mirror_status 2023-08-26T13:36:07.648 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ echo cluster1-mirror.mirror_status 2023-08-26T13:36:07.648 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ sed -e 's|[/ ]|_|g' 2023-08-26T13:36:07.649 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local status_log=/tmp/tmp.TfDn6BcQ3N/cluster1-mirror.mirror_status 2023-08-26T13:36:07.649 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2023-08-26T13:36:07.685 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:rbd: failed to query services: (13) Permission denied 2023-08-26T13:36:07.709 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local last_update state 2023-08-26T13:36:07.709 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update' 2023-08-26T13:36:07.711 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ last_update='2023-08-26 13:35:45' 2023-08-26T13:36:07.711 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state' 2023-08-26T13:36:07.713 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ state=up+replaying 2023-08-26T13:36:07.713 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ echo up+replaying 2023-08-26T13:36:07.713 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ grep up+replaying 2023-08-26T13:36:07.714 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:up+replaying 2023-08-26T13:36:07.715 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ return 0 2023-08-26T13:36:07.715 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ return 0 2023-08-26T13:36:07.715 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ snap_name=snap4 2023-08-26T13:36:07.715 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ create_snap cluster2 mirror test snap4 2023-08-26T13:36:07.716 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster2 2023-08-26T13:36:07.716 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T13:36:07.716 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=test 2023-08-26T13:36:07.716 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local snap_name=snap4 2023-08-26T13:36:07.716 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rbd --cluster cluster2 -p mirror snap create test@snap4 --debug-rbd=20 --debug-journaler=20 2023-08-26T13:36:25.458 INFO:tasks.rbd_mirror_thrash:kill cluster1.client.mirror.0 2023-08-26T13:36:25.459 INFO:tasks.rbd_mirror.cluster1.client.mirror.0:Sent signal 15 2023-08-26T13:36:25.459 INFO:tasks.rbd_mirror_thrash:waiting for 6 secs before reviving daemons 2023-08-26T13:36:25.459 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi124.stderr:2023-08-26T13:36:25.458+0000 7f04c094d700 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.0 (PID: 108761) UID: 1000 2023-08-26T13:36:31.460 INFO:tasks.rbd_mirror_thrash:waiting for cluster1.client.mirror.0 2023-08-26T13:36:31.461 INFO:teuthology.orchestra.run:waiting for 600 2023-08-26T13:36:31.461 INFO:tasks.rbd_mirror_thrash:reviving cluster1.client.mirror.0 2023-08-26T13:36:31.461 INFO:tasks.rbd_mirror.cluster1.client.mirror.0:Restarting daemon 2023-08-26T13:36:31.461 DEBUG:teuthology.orchestra.run.smithi124:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.0 2023-08-26T13:36:31.464 INFO:tasks.rbd_mirror.cluster1.client.mirror.0:Started 2023-08-26T13:36:31.464 INFO:tasks.rbd_mirror_thrash:waiting for 73 secs before thrashing 2023-08-26T13:36:34.641 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ wait_for_image_replay_started cluster1 mirror test 2023-08-26T13:36:34.642 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster1 2023-08-26T13:36:34.642 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T13:36:34.642 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=test 2023-08-26T13:36:34.642 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ wait_for_image_replay_state cluster1 mirror test started 2023-08-26T13:36:34.642 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster1 2023-08-26T13:36:34.643 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T13:36:34.643 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=test 2023-08-26T13:36:34.643 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local state=started 2023-08-26T13:36:34.643 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local s 2023-08-26T13:36:34.644 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-08-26T13:36:34.644 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ sleep 1 2023-08-26T13:36:35.643 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ test_image_replay_state cluster1 mirror test started 2023-08-26T13:36:35.643 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster1 2023-08-26T13:36:35.643 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T13:36:35.644 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=test 2023-08-26T13:36:35.644 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local test_state=started 2023-08-26T13:36:35.644 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local status_result 2023-08-26T13:36:35.644 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local current_state=stopped 2023-08-26T13:36:35.647 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ admin_daemons cluster1 rbd mirror status mirror/test 2023-08-26T13:36:35.647 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local cluster_instance=cluster1 2023-08-26T13:36:35.648 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ shift 2023-08-26T13:36:35.648 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ grep -i state 2023-08-26T13:36:35.648 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local cluster=cluster1 2023-08-26T13:36:35.648 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local instance=cluster1 2023-08-26T13:36:35.648 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local loop_instance 2023-08-26T13:36:35.648 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-08-26T13:36:35.649 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ sleep 0 2023-08-26T13:36:35.649 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ '[' cluster1 '!=' cluster1 ']' 2023-08-26T13:36:35.649 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ seq 0 3 2023-08-26T13:36:35.649 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-08-26T13:36:35.649 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test 2023-08-26T13:36:35.650 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local cluster=cluster1:0 2023-08-26T13:36:35.650 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ shift 2023-08-26T13:36:35.650 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local instance 2023-08-26T13:36:35.650 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ set_cluster_instance cluster1:0 cluster instance 2023-08-26T13:36:35.650 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local val=cluster1:0 2023-08-26T13:36:35.651 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local cluster_var_name=cluster 2023-08-26T13:36:35.651 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local instance_var_name=instance 2023-08-26T13:36:35.651 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ cluster=cluster1 2023-08-26T13:36:35.651 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ instance=0 2023-08-26T13:36:35.651 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ '[' 0 = cluster1:0 ']' 2023-08-26T13:36:35.652 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ eval cluster=cluster1 2023-08-26T13:36:35.652 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ cluster=cluster1 2023-08-26T13:36:35.652 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ eval instance=0 2023-08-26T13:36:35.652 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ instance=0 2023-08-26T13:36:35.652 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ daemon_asok_file cluster1:0 cluster1 2023-08-26T13:36:35.652 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ local local_cluster=cluster1:0 2023-08-26T13:36:35.653 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ local cluster=cluster1 2023-08-26T13:36:35.653 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ local instance 2023-08-26T13:36:35.653 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2023-08-26T13:36:35.653 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ local val=cluster1:0 2023-08-26T13:36:35.653 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ local cluster_var_name=local_cluster 2023-08-26T13:36:35.653 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ local instance_var_name=instance 2023-08-26T13:36:35.655 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ cluster=cluster1 2023-08-26T13:36:35.655 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ instance=0 2023-08-26T13:36:35.655 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ '[' 0 = cluster1:0 ']' 2023-08-26T13:36:35.655 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ eval local_cluster=cluster1 2023-08-26T13:36:35.655 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++++ local_cluster=cluster1 2023-08-26T13:36:35.656 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ eval instance=0 2023-08-26T13:36:35.656 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++++ instance=0 2023-08-26T13:36:35.656 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2023-08-26T13:36:35.670 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-08-26T13:36:35.670 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-08-26T13:36:35.670 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-08-26T13:36:35.670 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2023-08-26T13:36:35.770 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:Invalid command: unused arguments: ['mirror/test'] 2023-08-26T13:36:35.770 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:rbd mirror status : get status for rbd mirror 2023-08-26T13:36:35.770 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:admin_socket: invalid command 2023-08-26T13:36:35.773 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-08-26T13:36:35.774 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/test 2023-08-26T13:36:35.774 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local cluster=cluster1:1 2023-08-26T13:36:35.774 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ shift 2023-08-26T13:36:35.774 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local instance 2023-08-26T13:36:35.774 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ set_cluster_instance cluster1:1 cluster instance 2023-08-26T13:36:35.775 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local val=cluster1:1 2023-08-26T13:36:35.775 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local cluster_var_name=cluster 2023-08-26T13:36:35.775 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local instance_var_name=instance 2023-08-26T13:36:35.775 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ cluster=cluster1 2023-08-26T13:36:35.776 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ instance=1 2023-08-26T13:36:35.776 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ '[' 1 = cluster1:1 ']' 2023-08-26T13:36:35.776 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ eval cluster=cluster1 2023-08-26T13:36:35.776 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ cluster=cluster1 2023-08-26T13:36:35.776 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ eval instance=1 2023-08-26T13:36:35.777 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ instance=1 2023-08-26T13:36:35.777 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ daemon_asok_file cluster1:1 cluster1 2023-08-26T13:36:35.777 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ local local_cluster=cluster1:1 2023-08-26T13:36:35.777 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ local cluster=cluster1 2023-08-26T13:36:35.778 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ local instance 2023-08-26T13:36:35.778 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2023-08-26T13:36:35.778 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ local val=cluster1:1 2023-08-26T13:36:35.778 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ local cluster_var_name=local_cluster 2023-08-26T13:36:35.779 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ local instance_var_name=instance 2023-08-26T13:36:35.779 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ cluster=cluster1 2023-08-26T13:36:35.779 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ instance=1 2023-08-26T13:36:35.779 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ '[' 1 = cluster1:1 ']' 2023-08-26T13:36:35.779 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ eval local_cluster=cluster1 2023-08-26T13:36:35.780 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++++ local_cluster=cluster1 2023-08-26T13:36:35.780 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ eval instance=1 2023-08-26T13:36:35.780 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++++ instance=1 2023-08-26T13:36:35.780 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2023-08-26T13:36:35.792 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-08-26T13:36:35.792 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-08-26T13:36:35.792 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-08-26T13:36:35.792 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/test 2023-08-26T13:36:35.886 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:Invalid command: unused arguments: ['mirror/test'] 2023-08-26T13:36:35.886 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:rbd mirror status : get status for rbd mirror 2023-08-26T13:36:35.886 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:admin_socket: invalid command 2023-08-26T13:36:35.889 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-08-26T13:36:35.889 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/test 2023-08-26T13:36:35.889 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local cluster=cluster1:2 2023-08-26T13:36:35.889 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ shift 2023-08-26T13:36:35.889 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local instance 2023-08-26T13:36:35.890 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ set_cluster_instance cluster1:2 cluster instance 2023-08-26T13:36:35.890 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local val=cluster1:2 2023-08-26T13:36:35.890 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local cluster_var_name=cluster 2023-08-26T13:36:35.890 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local instance_var_name=instance 2023-08-26T13:36:35.890 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ cluster=cluster1 2023-08-26T13:36:35.891 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ instance=2 2023-08-26T13:36:35.891 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ '[' 2 = cluster1:2 ']' 2023-08-26T13:36:35.891 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ eval cluster=cluster1 2023-08-26T13:36:35.891 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ cluster=cluster1 2023-08-26T13:36:35.891 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ eval instance=2 2023-08-26T13:36:35.891 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ instance=2 2023-08-26T13:36:35.892 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ daemon_asok_file cluster1:2 cluster1 2023-08-26T13:36:35.892 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ local local_cluster=cluster1:2 2023-08-26T13:36:35.892 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ local cluster=cluster1 2023-08-26T13:36:35.892 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ local instance 2023-08-26T13:36:35.892 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2023-08-26T13:36:35.893 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ local val=cluster1:2 2023-08-26T13:36:35.893 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ local cluster_var_name=local_cluster 2023-08-26T13:36:35.893 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ local instance_var_name=instance 2023-08-26T13:36:35.893 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ cluster=cluster1 2023-08-26T13:36:35.893 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ instance=2 2023-08-26T13:36:35.894 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ '[' 2 = cluster1:2 ']' 2023-08-26T13:36:35.894 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ eval local_cluster=cluster1 2023-08-26T13:36:35.894 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++++ local_cluster=cluster1 2023-08-26T13:36:35.894 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ eval instance=2 2023-08-26T13:36:35.894 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++++ instance=2 2023-08-26T13:36:35.894 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2023-08-26T13:36:35.907 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-08-26T13:36:35.907 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-08-26T13:36:35.907 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-08-26T13:36:35.908 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/test 2023-08-26T13:36:36.011 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ return 0 2023-08-26T13:36:36.011 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ status_result=' "state": "Replaying"' 2023-08-26T13:36:36.012 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ echo ' "state": "Replaying"' 2023-08-26T13:36:36.012 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ grep -i Replaying 2023-08-26T13:36:36.012 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: "state": "Replaying" 2023-08-26T13:36:36.015 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ current_state=started 2023-08-26T13:36:36.015 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ test started = started 2023-08-26T13:36:36.015 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ return 0 2023-08-26T13:36:36.015 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror test 2023-08-26T13:36:36.015 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local local_cluster=cluster1 2023-08-26T13:36:36.016 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster2 2023-08-26T13:36:36.016 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T13:36:36.016 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=test 2023-08-26T13:36:36.016 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ '[' snapshot = journal ']' 2023-08-26T13:36:36.017 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ '[' snapshot = snapshot ']' 2023-08-26T13:36:36.017 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror test 2023-08-26T13:36:36.017 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local local_cluster=cluster1 2023-08-26T13:36:36.017 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster2 2023-08-26T13:36:36.017 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T13:36:36.017 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=test 2023-08-26T13:36:36.018 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ mkfname cluster2-mirror-test.status 2023-08-26T13:36:36.018 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ echo cluster2-mirror-test.status 2023-08-26T13:36:36.018 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ sed -e 's|[/ ]|_|g' 2023-08-26T13:36:36.018 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local status_log=/tmp/tmp.TfDn6BcQ3N/cluster2-mirror-test.status 2023-08-26T13:36:36.018 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ mkfname cluster1-mirror-test.status 2023-08-26T13:36:36.019 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ echo cluster1-mirror-test.status 2023-08-26T13:36:36.019 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ sed -e 's|[/ ]|_|g' 2023-08-26T13:36:36.020 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local local_status_log=/tmp/tmp.TfDn6BcQ3N/cluster1-mirror-test.status 2023-08-26T13:36:36.020 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ mirror_image_snapshot cluster2 mirror test 2023-08-26T13:36:36.020 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster2 2023-08-26T13:36:36.020 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T13:36:36.020 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=test 2023-08-26T13:36:36.021 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/test 2023-08-26T13:36:37.637 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:Snapshot ID: 17 2023-08-26T13:36:37.651 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ get_newest_mirror_snapshot cluster2 mirror test /tmp/tmp.TfDn6BcQ3N/cluster2-mirror-test.status 2023-08-26T13:36:37.652 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster2 2023-08-26T13:36:37.652 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T13:36:37.652 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=test 2023-08-26T13:36:37.652 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local log=/tmp/tmp.TfDn6BcQ3N/cluster2-mirror-test.status 2023-08-26T13:36:37.654 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rbd --cluster cluster2 snap list --all mirror/test --format xml 2023-08-26T13:36:37.655 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-08-26T13:36:37.714 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ xmlstarlet sel -t -v //snapshot/id 2023-08-26T13:36:37.716 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local snapshot_id=17 2023-08-26T13:36:37.716 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ true 2023-08-26T13:36:37.717 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-08-26T13:36:37.717 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ sleep 0.2 2023-08-26T13:36:37.918 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.TfDn6BcQ3N/cluster1-mirror-test.status 2023-08-26T13:36:37.918 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster1 2023-08-26T13:36:37.918 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T13:36:37.918 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=test 2023-08-26T13:36:37.919 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local log=/tmp/tmp.TfDn6BcQ3N/cluster1-mirror-test.status 2023-08-26T13:36:37.919 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2023-08-26T13:36:37.919 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-08-26T13:36:37.966 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ true 2023-08-26T13:36:37.967 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-08-26T13:36:37.968 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:-:1.1: Document is empty 2023-08-26T13:36:37.969 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr: 2023-08-26T13:36:37.969 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:^ 2023-08-26T13:36:37.969 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local primary_snapshot_id= 2023-08-26T13:36:37.969 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ test 17 = '' 2023-08-26T13:36:37.969 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-08-26T13:36:37.970 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ sleep 0.4 2023-08-26T13:36:38.369 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.TfDn6BcQ3N/cluster1-mirror-test.status 2023-08-26T13:36:38.370 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster1 2023-08-26T13:36:38.370 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T13:36:38.370 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=test 2023-08-26T13:36:38.370 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local log=/tmp/tmp.TfDn6BcQ3N/cluster1-mirror-test.status 2023-08-26T13:36:38.370 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2023-08-26T13:36:38.371 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-08-26T13:36:38.418 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ true 2023-08-26T13:36:38.419 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-08-26T13:36:38.420 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:-:1.1: Document is empty 2023-08-26T13:36:38.420 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr: 2023-08-26T13:36:38.420 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:^ 2023-08-26T13:36:38.421 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local primary_snapshot_id= 2023-08-26T13:36:38.421 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ test 17 = '' 2023-08-26T13:36:38.421 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-08-26T13:36:38.421 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ sleep 0.8 2023-08-26T13:36:39.222 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.TfDn6BcQ3N/cluster1-mirror-test.status 2023-08-26T13:36:39.222 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster1 2023-08-26T13:36:39.222 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T13:36:39.222 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=test 2023-08-26T13:36:39.223 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local log=/tmp/tmp.TfDn6BcQ3N/cluster1-mirror-test.status 2023-08-26T13:36:39.225 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2023-08-26T13:36:39.226 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-08-26T13:36:39.285 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ true 2023-08-26T13:36:39.285 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-08-26T13:36:39.286 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:-:1.1: Document is empty 2023-08-26T13:36:39.286 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr: 2023-08-26T13:36:39.286 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:^ 2023-08-26T13:36:39.287 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local primary_snapshot_id= 2023-08-26T13:36:39.287 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ test 17 = '' 2023-08-26T13:36:39.287 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-08-26T13:36:39.287 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ sleep 1.6 2023-08-26T13:36:40.888 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.TfDn6BcQ3N/cluster1-mirror-test.status 2023-08-26T13:36:40.888 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster1 2023-08-26T13:36:40.888 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T13:36:40.889 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=test 2023-08-26T13:36:40.889 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local log=/tmp/tmp.TfDn6BcQ3N/cluster1-mirror-test.status 2023-08-26T13:36:40.889 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2023-08-26T13:36:40.889 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-08-26T13:36:40.937 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-08-26T13:36:40.939 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local primary_snapshot_id=17 2023-08-26T13:36:40.939 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ test 17 = 17 2023-08-26T13:36:40.939 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ return 0 2023-08-26T13:36:40.940 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ wait_for_snap_present cluster1 mirror test snap4 2023-08-26T13:36:40.940 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster1 2023-08-26T13:36:40.940 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T13:36:40.940 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=test 2023-08-26T13:36:40.940 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local snap_name=snap4 2023-08-26T13:36:40.941 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local s 2023-08-26T13:36:40.941 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2023-08-26T13:36:40.941 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ sleep 1 2023-08-26T13:36:41.940 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rbd --cluster cluster1 info mirror/test@snap4 2023-08-26T13:36:41.987 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:rbd image 'test': 2023-08-26T13:36:41.987 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: size 512 MiB in 128 objects 2023-08-26T13:36:41.987 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: order 22 (4 MiB objects) 2023-08-26T13:36:41.987 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: snapshot_count: 8 2023-08-26T13:36:41.988 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: id: 10ab632cdafc 2023-08-26T13:36:41.988 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: block_name_prefix: rbd_data.10ab632cdafc 2023-08-26T13:36:41.988 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: format: 2 2023-08-26T13:36:41.988 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: features: layering, exclusive-lock, non-primary 2023-08-26T13:36:41.988 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: op_features: 2023-08-26T13:36:41.989 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: flags: 2023-08-26T13:36:41.989 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: create_timestamp: Sat Aug 26 13:33:17 2023 2023-08-26T13:36:41.989 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: access_timestamp: Sat Aug 26 13:35:55 2023 2023-08-26T13:36:41.989 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: modify_timestamp: Sat Aug 26 13:33:17 2023 2023-08-26T13:36:41.989 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: protected: False 2023-08-26T13:36:41.990 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: mirroring state: enabled 2023-08-26T13:36:41.990 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: mirroring mode: snapshot 2023-08-26T13:36:41.990 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: mirroring global id: e2f2554d-659e-4467-b7f8-dddd7ccb0ee9 2023-08-26T13:36:41.990 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: mirroring primary: false 2023-08-26T13:36:41.991 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ return 0 2023-08-26T13:36:41.992 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ '[' -n snap3-clean ']' 2023-08-26T13:36:41.992 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ compare_image_snaps mirror test snap3-clean 2023-08-26T13:36:41.992 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T13:36:41.992 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=test 2023-08-26T13:36:41.992 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local snap_name=snap3-clean 2023-08-26T13:36:41.993 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local ret=0 2023-08-26T13:36:41.993 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local rmt_export=/tmp/tmp.TfDn6BcQ3N/cluster2-mirror-test.export 2023-08-26T13:36:41.993 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local loc_export=/tmp/tmp.TfDn6BcQ3N/cluster1-mirror-test.export 2023-08-26T13:36:41.993 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rm -f /tmp/tmp.TfDn6BcQ3N/cluster2-mirror-test.export /tmp/tmp.TfDn6BcQ3N/cluster1-mirror-test.export 2023-08-26T13:36:41.993 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rbd --cluster cluster2 -p mirror export test@snap3-clean /tmp/tmp.TfDn6BcQ3N/cluster2-mirror-test.export 2023-08-26T13:36:42.571 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2023-08-26T13:36:42.582 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rbd --cluster cluster1 -p mirror export test@snap3-clean /tmp/tmp.TfDn6BcQ3N/cluster1-mirror-test.export 2023-08-26T13:36:43.204 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2023-08-26T13:36:43.212 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ cmp /tmp/tmp.TfDn6BcQ3N/cluster2-mirror-test.export /tmp/tmp.TfDn6BcQ3N/cluster1-mirror-test.export 2023-08-26T13:36:43.422 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rm -f /tmp/tmp.TfDn6BcQ3N/cluster2-mirror-test.export /tmp/tmp.TfDn6BcQ3N/cluster1-mirror-test.export 2023-08-26T13:36:43.569 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ return 0 2023-08-26T13:36:43.570 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ compare_image_snaps mirror test snap4 2023-08-26T13:36:43.570 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T13:36:43.570 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=test 2023-08-26T13:36:43.570 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local snap_name=snap4 2023-08-26T13:36:43.570 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local ret=0 2023-08-26T13:36:43.571 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local rmt_export=/tmp/tmp.TfDn6BcQ3N/cluster2-mirror-test.export 2023-08-26T13:36:43.571 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local loc_export=/tmp/tmp.TfDn6BcQ3N/cluster1-mirror-test.export 2023-08-26T13:36:43.571 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rm -f /tmp/tmp.TfDn6BcQ3N/cluster2-mirror-test.export /tmp/tmp.TfDn6BcQ3N/cluster1-mirror-test.export 2023-08-26T13:36:43.571 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rbd --cluster cluster2 -p mirror export test@snap4 /tmp/tmp.TfDn6BcQ3N/cluster2-mirror-test.export 2023-08-26T13:36:44.153 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2023-08-26T13:36:44.161 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rbd --cluster cluster1 -p mirror export test@snap4 /tmp/tmp.TfDn6BcQ3N/cluster1-mirror-test.export 2023-08-26T13:36:44.876 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2023-08-26T13:36:44.884 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ cmp /tmp/tmp.TfDn6BcQ3N/cluster2-mirror-test.export /tmp/tmp.TfDn6BcQ3N/cluster1-mirror-test.export 2023-08-26T13:36:45.096 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rm -f /tmp/tmp.TfDn6BcQ3N/cluster2-mirror-test.export /tmp/tmp.TfDn6BcQ3N/cluster1-mirror-test.export 2023-08-26T13:36:45.240 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ return 0 2023-08-26T13:36:45.241 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ clean_snap_name=snap4-clean 2023-08-26T13:36:45.241 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ create_snap cluster2 mirror test snap4-clean 2023-08-26T13:36:45.241 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster2 2023-08-26T13:36:45.241 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T13:36:45.241 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=test 2023-08-26T13:36:45.242 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local snap_name=snap4-clean 2023-08-26T13:36:45.242 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rbd --cluster cluster2 -p mirror snap create test@snap4-clean --debug-rbd=20 --debug-journaler=20 2023-08-26T13:36:45.590 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ for i in `seq 1 10` 2023-08-26T13:36:45.590 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ stress_write_image cluster2 mirror test 2023-08-26T13:36:45.590 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster2 2023-08-26T13:36:45.591 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T13:36:45.591 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=test 2023-08-26T13:36:45.591 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2023-08-26T13:36:45.592 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local duration=14 2023-08-26T13:36:45.592 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ set +e 2023-08-26T13:36:45.592 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ timeout 14s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror test --debug-rbd=20 --debug-journaler=20 2023-08-26T13:36:45.779 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:100 IOs, 14911488 bytes 2023-08-26T13:36:45.944 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:200 IOs, 50009088 bytes 2023-08-26T13:36:46.117 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:300 IOs, 101587968 bytes 2023-08-26T13:36:46.251 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:400 IOs, 136163328 bytes 2023-08-26T13:36:46.416 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:500 IOs, 198694912 bytes 2023-08-26T13:36:46.589 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:600 IOs, 234570752 bytes 2023-08-26T13:36:46.666 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:700 IOs, 235175936 bytes 2023-08-26T13:36:46.794 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:800 IOs, 263337984 bytes 2023-08-26T13:36:46.972 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:900 IOs, 308342784 bytes 2023-08-26T13:36:47.219 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:1000 IOs, 363599872 bytes 2023-08-26T13:36:47.337 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:1100 IOs, 387470336 bytes 2023-08-26T13:36:47.414 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:1200 IOs, 390932480 bytes 2023-08-26T13:36:47.515 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:1300 IOs, 404353024 bytes 2023-08-26T13:36:47.646 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:1400 IOs, 451812352 bytes 2023-08-26T13:36:47.881 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:1500 IOs, 525948928 bytes 2023-08-26T13:36:47.988 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:1600 IOs, 544678912 bytes 2023-08-26T13:36:48.305 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:1700 IOs, 679179264 bytes 2023-08-26T13:36:48.508 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:1800 IOs, 748711936 bytes 2023-08-26T13:36:48.572 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:1900 IOs, 749324288 bytes 2023-08-26T13:36:48.772 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:2000 IOs, 811646976 bytes 2023-08-26T13:36:48.895 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:2100 IOs, 833702912 bytes 2023-08-26T13:36:48.986 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:2200 IOs, 843852800 bytes 2023-08-26T13:36:49.144 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:2300 IOs, 881463296 bytes 2023-08-26T13:36:49.233 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:2400 IOs, 889113600 bytes 2023-08-26T13:36:49.341 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:2500 IOs, 905968640 bytes 2023-08-26T13:36:49.427 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:2600 IOs, 913939456 bytes 2023-08-26T13:36:49.531 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:2700 IOs, 929248256 bytes 2023-08-26T13:36:49.600 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:2800 IOs, 934181888 bytes 2023-08-26T13:36:49.971 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:2900 IOs, 1082189824 bytes 2023-08-26T13:36:50.089 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:3000 IOs, 1107630080 bytes 2023-08-26T13:36:50.226 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:3100 IOs, 1137275904 bytes 2023-08-26T13:36:50.346 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:3200 IOs, 1162599424 bytes 2023-08-26T13:36:50.427 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:3300 IOs, 1169387520 bytes 2023-08-26T13:36:50.578 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:3400 IOs, 1219730432 bytes 2023-08-26T13:36:50.891 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:3500 IOs, 1334444032 bytes 2023-08-26T13:36:51.018 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:3600 IOs, 1368282112 bytes 2023-08-26T13:36:51.103 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:3700 IOs, 1372814336 bytes 2023-08-26T13:36:51.238 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:3800 IOs, 1410063360 bytes 2023-08-26T13:36:51.415 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:3900 IOs, 1457740800 bytes 2023-08-26T13:36:51.595 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:4000 IOs, 1517178880 bytes 2023-08-26T13:36:51.768 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:4100 IOs, 1565950976 bytes 2023-08-26T13:36:51.841 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:4200 IOs, 1576872960 bytes 2023-08-26T13:36:52.014 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:4300 IOs, 1629963264 bytes 2023-08-26T13:36:52.148 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:4400 IOs, 1657624576 bytes 2023-08-26T13:36:52.303 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:4500 IOs, 1703440384 bytes 2023-08-26T13:36:52.586 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:4600 IOs, 1801759744 bytes 2023-08-26T13:36:52.670 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:4700 IOs, 1810864128 bytes 2023-08-26T13:36:52.823 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:4800 IOs, 1849133056 bytes 2023-08-26T13:36:52.943 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:4900 IOs, 1872807936 bytes 2023-08-26T13:36:53.091 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:5000 IOs, 1919463424 bytes 2023-08-26T13:36:53.276 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:5100 IOs, 1977516032 bytes 2023-08-26T13:36:53.541 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:5200 IOs, 2079686656 bytes 2023-08-26T13:36:53.687 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:5300 IOs, 2095887360 bytes 2023-08-26T13:36:53.862 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:5400 IOs, 2149376000 bytes 2023-08-26T13:36:53.954 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:5500 IOs, 2162580480 bytes 2023-08-26T13:36:54.104 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:5600 IOs, 2202184704 bytes 2023-08-26T13:36:54.232 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:5700 IOs, 2235789312 bytes 2023-08-26T13:36:54.335 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:5800 IOs, 2251195392 bytes 2023-08-26T13:36:54.553 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:5900 IOs, 2318793728 bytes 2023-08-26T13:36:54.648 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:6000 IOs, 2327881728 bytes 2023-08-26T13:36:54.815 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:6100 IOs, 2379476992 bytes 2023-08-26T13:36:55.041 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:6200 IOs, 2455800832 bytes 2023-08-26T13:36:55.207 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:6300 IOs, 2505460736 bytes 2023-08-26T13:36:55.383 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:6400 IOs, 2570705920 bytes 2023-08-26T13:36:55.499 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:6500 IOs, 2590161920 bytes 2023-08-26T13:36:55.572 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:6600 IOs, 2590758912 bytes 2023-08-26T13:36:55.784 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:6700 IOs, 2670635008 bytes 2023-08-26T13:36:55.907 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:6800 IOs, 2698759168 bytes 2023-08-26T13:36:56.034 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:6900 IOs, 2731749376 bytes 2023-08-26T13:36:56.118 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:7000 IOs, 2732339200 bytes 2023-08-26T13:36:56.276 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:7100 IOs, 2774795264 bytes 2023-08-26T13:36:56.383 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:7200 IOs, 2793318400 bytes 2023-08-26T13:36:56.647 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:7300 IOs, 2903247872 bytes 2023-08-26T13:36:56.810 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:7400 IOs, 2930037760 bytes 2023-08-26T13:36:56.912 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:7500 IOs, 2947861504 bytes 2023-08-26T13:36:57.027 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:7600 IOs, 2964189184 bytes 2023-08-26T13:36:57.165 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:7700 IOs, 3022158848 bytes 2023-08-26T13:36:57.318 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:7800 IOs, 3067465728 bytes 2023-08-26T13:36:57.498 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:7900 IOs, 3123930112 bytes 2023-08-26T13:36:57.595 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:8000 IOs, 3132380160 bytes 2023-08-26T13:36:57.780 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:8100 IOs, 3192968192 bytes 2023-08-26T13:36:57.941 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:8200 IOs, 3229691904 bytes 2023-08-26T13:36:58.027 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:8300 IOs, 3256196096 bytes 2023-08-26T13:36:58.248 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:8400 IOs, 3307501568 bytes 2023-08-26T13:36:58.423 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:8500 IOs, 3357175808 bytes 2023-08-26T13:36:58.788 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:8600 IOs, 3509667840 bytes 2023-08-26T13:36:58.929 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:8700 IOs, 3542977536 bytes 2023-08-26T13:36:59.038 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:8800 IOs, 3560804352 bytes 2023-08-26T13:36:59.156 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:8900 IOs, 3584083968 bytes 2023-08-26T13:36:59.353 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:9000 IOs, 3644490752 bytes 2023-08-26T13:36:59.480 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:9100 IOs, 3664711680 bytes 2023-08-26T13:36:59.590 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:9200 IOs, 3679568896 bytes 2023-08-26T13:36:59.596 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ error_code=124 2023-08-26T13:36:59.596 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ set -e 2023-08-26T13:36:59.596 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ '[' 124 -eq 124 ']' 2023-08-26T13:36:59.596 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ return 0 2023-08-26T13:36:59.596 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ wait_for_status_in_pool_dir cluster1 mirror test up+replaying 2023-08-26T13:36:59.597 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster1 2023-08-26T13:36:59.597 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T13:36:59.597 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=test 2023-08-26T13:36:59.597 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local state_pattern=up+replaying 2023-08-26T13:36:59.597 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local description_pattern= 2023-08-26T13:36:59.598 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local service_pattern= 2023-08-26T13:36:59.598 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-08-26T13:36:59.598 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ sleep 1 2023-08-26T13:37:00.597 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ test_status_in_pool_dir cluster1 mirror test up+replaying '' '' 2023-08-26T13:37:00.598 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster1 2023-08-26T13:37:00.598 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T13:37:00.598 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=test 2023-08-26T13:37:00.598 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local state_pattern=up+replaying 2023-08-26T13:37:00.598 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local description_pattern= 2023-08-26T13:37:00.599 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local service_pattern= 2023-08-26T13:37:00.599 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ mkfname cluster1-mirror-test.mirror_status 2023-08-26T13:37:00.599 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ echo cluster1-mirror-test.mirror_status 2023-08-26T13:37:00.599 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ sed -e 's|[/ ]|_|g' 2023-08-26T13:37:00.600 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local status_log=/tmp/tmp.TfDn6BcQ3N/cluster1-mirror-test.mirror_status 2023-08-26T13:37:00.600 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ CEPH_ARGS= 2023-08-26T13:37:00.600 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rbd --cluster cluster1 mirror image status mirror/test 2023-08-26T13:37:00.600 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ tee /tmp/tmp.TfDn6BcQ3N/cluster1-mirror-test.mirror_status 2023-08-26T13:37:00.650 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:test: 2023-08-26T13:37:00.650 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr: global_id: e2f2554d-659e-4467-b7f8-dddd7ccb0ee9 2023-08-26T13:37:00.651 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr: state: up+replaying 2023-08-26T13:37:00.651 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr: description: replaying, {"bytes_per_second":17730013.87,"bytes_per_snapshot":524978688.0,"last_snapshot_bytes":531900416,"last_snapshot_sync_seconds":1,"local_snapshot_timestamp":1693056996,"remote_snapshot_timestamp":1693056996,"replay_state":"idle"} 2023-08-26T13:37:00.651 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr: service: mirror.2 on smithi124 2023-08-26T13:37:00.652 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr: last_update: 2023-08-26 13:36:45 2023-08-26T13:37:00.652 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr: peer_sites: 2023-08-26T13:37:00.652 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr: name: cluster2 2023-08-26T13:37:00.652 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr: state: up+stopped 2023-08-26T13:37:00.653 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr: description: local image is primary 2023-08-26T13:37:00.653 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr: last_update: 2023-08-26 13:36:45 2023-08-26T13:37:00.655 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ grep '^ state: .*up+replaying' /tmp/tmp.TfDn6BcQ3N/cluster1-mirror-test.mirror_status 2023-08-26T13:37:00.656 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: state: up+replaying 2023-08-26T13:37:00.656 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ grep '^ description: .*' /tmp/tmp.TfDn6BcQ3N/cluster1-mirror-test.mirror_status 2023-08-26T13:37:00.657 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: description: replaying, {"bytes_per_second":17730013.87,"bytes_per_snapshot":524978688.0,"last_snapshot_bytes":531900416,"last_snapshot_sync_seconds":1,"local_snapshot_timestamp":1693056996,"remote_snapshot_timestamp":1693056996,"replay_state":"idle"} 2023-08-26T13:37:00.657 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ '[' -n '' ']' 2023-08-26T13:37:00.658 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ echo up+replaying 2023-08-26T13:37:00.658 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ grep '^up+' 2023-08-26T13:37:00.658 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:up+replaying 2023-08-26T13:37:00.659 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ grep 'service: *mirror..* on ' /tmp/tmp.TfDn6BcQ3N/cluster1-mirror-test.mirror_status 2023-08-26T13:37:00.659 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: service: mirror.2 on smithi124 2023-08-26T13:37:00.660 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' /tmp/tmp.TfDn6BcQ3N/cluster1-mirror-test.mirror_status 2023-08-26T13:37:00.662 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local 'last_update=2023-08-26 13:36:45' 2023-08-26T13:37:00.663 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ test_mirror_pool_status_verbose cluster1 mirror test up+replaying '2023-08-26 13:36:45' 2023-08-26T13:37:00.663 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster1 2023-08-26T13:37:00.663 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T13:37:00.664 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=test 2023-08-26T13:37:00.664 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local state_pattern=up+replaying 2023-08-26T13:37:00.664 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local 'prev_last_update=2023-08-26 13:36:45' 2023-08-26T13:37:00.664 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ mkfname cluster1-mirror.mirror_status 2023-08-26T13:37:00.665 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ echo cluster1-mirror.mirror_status 2023-08-26T13:37:00.665 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ sed -e 's|[/ ]|_|g' 2023-08-26T13:37:00.665 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local status_log=/tmp/tmp.TfDn6BcQ3N/cluster1-mirror.mirror_status 2023-08-26T13:37:00.666 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2023-08-26T13:37:00.700 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:rbd: failed to query services: (13) Permission denied 2023-08-26T13:37:00.722 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local last_update state 2023-08-26T13:37:00.722 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update' 2023-08-26T13:37:00.723 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ last_update='2023-08-26 13:36:45' 2023-08-26T13:37:00.724 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state' 2023-08-26T13:37:00.725 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ state=up+replaying 2023-08-26T13:37:00.726 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ echo up+replaying 2023-08-26T13:37:00.726 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ grep up+replaying 2023-08-26T13:37:00.726 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:up+replaying 2023-08-26T13:37:00.728 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ return 0 2023-08-26T13:37:00.728 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ return 0 2023-08-26T13:37:00.728 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ snap_name=snap5 2023-08-26T13:37:00.729 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ create_snap cluster2 mirror test snap5 2023-08-26T13:37:00.729 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster2 2023-08-26T13:37:00.729 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T13:37:00.729 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=test 2023-08-26T13:37:00.729 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local snap_name=snap5 2023-08-26T13:37:00.729 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rbd --cluster cluster2 -p mirror snap create test@snap5 --debug-rbd=20 --debug-journaler=20 2023-08-26T13:37:11.453 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.0 with skip (0.8456049641003627) > weight (0.25) 2023-08-26T13:37:11.454 INFO:tasks.rbd_mirror_thrash:kill cluster2.client.mirror.1 2023-08-26T13:37:11.454 INFO:tasks.rbd_mirror.cluster2.client.mirror.1:Sent signal 15 2023-08-26T13:37:11.455 INFO:tasks.rbd_mirror_thrash:waiting for 1 secs before reviving daemons 2023-08-26T13:37:11.456 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi124.stderr:2023-08-26T13:37:11.454+0000 7fcb8be74700 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.1 (PID: 106076) UID: 1000 2023-08-26T13:37:12.456 INFO:tasks.rbd_mirror_thrash:waiting for cluster2.client.mirror.1 2023-08-26T13:37:12.457 INFO:teuthology.orchestra.run:waiting for 600 2023-08-26T13:37:12.457 INFO:tasks.rbd_mirror_thrash:reviving cluster2.client.mirror.1 2023-08-26T13:37:12.457 INFO:tasks.rbd_mirror.cluster2.client.mirror.1:Restarting daemon 2023-08-26T13:37:12.458 DEBUG:teuthology.orchestra.run.smithi124:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.1 2023-08-26T13:37:12.460 INFO:tasks.rbd_mirror.cluster2.client.mirror.1:Started 2023-08-26T13:37:12.461 INFO:tasks.rbd_mirror_thrash:waiting for 67 secs before thrashing 2023-08-26T13:37:30.625 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ wait_for_image_replay_started cluster1 mirror test 2023-08-26T13:37:30.625 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster1 2023-08-26T13:37:30.626 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T13:37:30.626 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=test 2023-08-26T13:37:30.626 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ wait_for_image_replay_state cluster1 mirror test started 2023-08-26T13:37:30.626 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster1 2023-08-26T13:37:30.626 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T13:37:30.627 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=test 2023-08-26T13:37:30.627 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local state=started 2023-08-26T13:37:30.627 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local s 2023-08-26T13:37:30.627 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-08-26T13:37:30.627 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ sleep 1 2023-08-26T13:37:31.627 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ test_image_replay_state cluster1 mirror test started 2023-08-26T13:37:31.627 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster1 2023-08-26T13:37:31.627 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T13:37:31.628 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=test 2023-08-26T13:37:31.628 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local test_state=started 2023-08-26T13:37:31.628 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local status_result 2023-08-26T13:37:31.628 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local current_state=stopped 2023-08-26T13:37:31.628 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ admin_daemons cluster1 rbd mirror status mirror/test 2023-08-26T13:37:31.629 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local cluster_instance=cluster1 2023-08-26T13:37:31.629 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ shift 2023-08-26T13:37:31.629 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local cluster=cluster1 2023-08-26T13:37:31.629 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local instance=cluster1 2023-08-26T13:37:31.630 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local loop_instance 2023-08-26T13:37:31.630 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ grep -i state 2023-08-26T13:37:31.630 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-08-26T13:37:31.630 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ sleep 0 2023-08-26T13:37:31.633 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ '[' cluster1 '!=' cluster1 ']' 2023-08-26T13:37:31.633 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ seq 0 3 2023-08-26T13:37:31.633 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-08-26T13:37:31.634 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test 2023-08-26T13:37:31.634 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local cluster=cluster1:0 2023-08-26T13:37:31.634 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ shift 2023-08-26T13:37:31.634 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local instance 2023-08-26T13:37:31.634 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ set_cluster_instance cluster1:0 cluster instance 2023-08-26T13:37:31.635 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local val=cluster1:0 2023-08-26T13:37:31.635 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local cluster_var_name=cluster 2023-08-26T13:37:31.635 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local instance_var_name=instance 2023-08-26T13:37:31.635 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ cluster=cluster1 2023-08-26T13:37:31.635 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ instance=0 2023-08-26T13:37:31.636 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ '[' 0 = cluster1:0 ']' 2023-08-26T13:37:31.636 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ eval cluster=cluster1 2023-08-26T13:37:31.636 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ cluster=cluster1 2023-08-26T13:37:31.636 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ eval instance=0 2023-08-26T13:37:31.636 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ instance=0 2023-08-26T13:37:31.637 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ daemon_asok_file cluster1:0 cluster1 2023-08-26T13:37:31.637 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ local local_cluster=cluster1:0 2023-08-26T13:37:31.637 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ local cluster=cluster1 2023-08-26T13:37:31.637 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ local instance 2023-08-26T13:37:31.637 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2023-08-26T13:37:31.638 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ local val=cluster1:0 2023-08-26T13:37:31.638 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ local cluster_var_name=local_cluster 2023-08-26T13:37:31.638 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ local instance_var_name=instance 2023-08-26T13:37:31.638 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ cluster=cluster1 2023-08-26T13:37:31.638 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ instance=0 2023-08-26T13:37:31.638 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ '[' 0 = cluster1:0 ']' 2023-08-26T13:37:31.639 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ eval local_cluster=cluster1 2023-08-26T13:37:31.639 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++++ local_cluster=cluster1 2023-08-26T13:37:31.639 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ eval instance=0 2023-08-26T13:37:31.639 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++++ instance=0 2023-08-26T13:37:31.639 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2023-08-26T13:37:31.653 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-08-26T13:37:31.653 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-08-26T13:37:31.653 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-08-26T13:37:31.653 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2023-08-26T13:37:31.754 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:Invalid command: unused arguments: ['mirror/test'] 2023-08-26T13:37:31.755 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:rbd mirror status : get status for rbd mirror 2023-08-26T13:37:31.755 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:admin_socket: invalid command 2023-08-26T13:37:31.758 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-08-26T13:37:31.758 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/test 2023-08-26T13:37:31.758 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local cluster=cluster1:1 2023-08-26T13:37:31.758 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ shift 2023-08-26T13:37:31.759 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local instance 2023-08-26T13:37:31.759 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ set_cluster_instance cluster1:1 cluster instance 2023-08-26T13:37:31.759 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local val=cluster1:1 2023-08-26T13:37:31.759 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local cluster_var_name=cluster 2023-08-26T13:37:31.759 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local instance_var_name=instance 2023-08-26T13:37:31.760 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ cluster=cluster1 2023-08-26T13:37:31.760 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ instance=1 2023-08-26T13:37:31.760 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ '[' 1 = cluster1:1 ']' 2023-08-26T13:37:31.760 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ eval cluster=cluster1 2023-08-26T13:37:31.760 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ cluster=cluster1 2023-08-26T13:37:31.761 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ eval instance=1 2023-08-26T13:37:31.761 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ instance=1 2023-08-26T13:37:31.761 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ daemon_asok_file cluster1:1 cluster1 2023-08-26T13:37:31.765 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ local local_cluster=cluster1:1 2023-08-26T13:37:31.765 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ local cluster=cluster1 2023-08-26T13:37:31.765 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ local instance 2023-08-26T13:37:31.766 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2023-08-26T13:37:31.766 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ local val=cluster1:1 2023-08-26T13:37:31.766 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ local cluster_var_name=local_cluster 2023-08-26T13:37:31.766 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ local instance_var_name=instance 2023-08-26T13:37:31.766 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ cluster=cluster1 2023-08-26T13:37:31.766 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ instance=1 2023-08-26T13:37:31.767 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ '[' 1 = cluster1:1 ']' 2023-08-26T13:37:31.767 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ eval local_cluster=cluster1 2023-08-26T13:37:31.767 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++++ local_cluster=cluster1 2023-08-26T13:37:31.767 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ eval instance=1 2023-08-26T13:37:31.767 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++++ instance=1 2023-08-26T13:37:31.768 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2023-08-26T13:37:31.776 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-08-26T13:37:31.777 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-08-26T13:37:31.777 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-08-26T13:37:31.777 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/test 2023-08-26T13:37:31.871 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:Invalid command: unused arguments: ['mirror/test'] 2023-08-26T13:37:31.872 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:rbd mirror status : get status for rbd mirror 2023-08-26T13:37:31.872 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:admin_socket: invalid command 2023-08-26T13:37:31.875 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-08-26T13:37:31.875 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/test 2023-08-26T13:37:31.875 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local cluster=cluster1:2 2023-08-26T13:37:31.875 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ shift 2023-08-26T13:37:31.876 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local instance 2023-08-26T13:37:31.876 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ set_cluster_instance cluster1:2 cluster instance 2023-08-26T13:37:31.876 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local val=cluster1:2 2023-08-26T13:37:31.876 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local cluster_var_name=cluster 2023-08-26T13:37:31.876 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local instance_var_name=instance 2023-08-26T13:37:31.876 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ cluster=cluster1 2023-08-26T13:37:31.877 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ instance=2 2023-08-26T13:37:31.877 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ '[' 2 = cluster1:2 ']' 2023-08-26T13:37:31.877 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ eval cluster=cluster1 2023-08-26T13:37:31.877 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ cluster=cluster1 2023-08-26T13:37:31.877 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ eval instance=2 2023-08-26T13:37:31.878 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ instance=2 2023-08-26T13:37:31.878 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ daemon_asok_file cluster1:2 cluster1 2023-08-26T13:37:31.878 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ local local_cluster=cluster1:2 2023-08-26T13:37:31.878 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ local cluster=cluster1 2023-08-26T13:37:31.879 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ local instance 2023-08-26T13:37:31.879 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2023-08-26T13:37:31.879 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ local val=cluster1:2 2023-08-26T13:37:31.879 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ local cluster_var_name=local_cluster 2023-08-26T13:37:31.879 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ local instance_var_name=instance 2023-08-26T13:37:31.879 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ cluster=cluster1 2023-08-26T13:37:31.880 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ instance=2 2023-08-26T13:37:31.880 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ '[' 2 = cluster1:2 ']' 2023-08-26T13:37:31.880 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ eval local_cluster=cluster1 2023-08-26T13:37:31.880 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++++ local_cluster=cluster1 2023-08-26T13:37:31.880 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ eval instance=2 2023-08-26T13:37:31.881 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++++ instance=2 2023-08-26T13:37:31.881 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2023-08-26T13:37:31.893 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-08-26T13:37:31.893 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-08-26T13:37:31.893 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-08-26T13:37:31.894 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/test 2023-08-26T13:37:31.997 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ return 0 2023-08-26T13:37:31.998 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ status_result=' "state": "Replaying"' 2023-08-26T13:37:31.998 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ echo ' "state": "Replaying"' 2023-08-26T13:37:31.998 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ grep -i Replaying 2023-08-26T13:37:31.999 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: "state": "Replaying" 2023-08-26T13:37:32.001 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ current_state=started 2023-08-26T13:37:32.001 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ test started = started 2023-08-26T13:37:32.001 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ return 0 2023-08-26T13:37:32.002 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror test 2023-08-26T13:37:32.002 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local local_cluster=cluster1 2023-08-26T13:37:32.002 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster2 2023-08-26T13:37:32.002 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T13:37:32.002 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=test 2023-08-26T13:37:32.003 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ '[' snapshot = journal ']' 2023-08-26T13:37:32.003 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ '[' snapshot = snapshot ']' 2023-08-26T13:37:32.003 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror test 2023-08-26T13:37:32.003 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local local_cluster=cluster1 2023-08-26T13:37:32.003 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster2 2023-08-26T13:37:32.004 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T13:37:32.004 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=test 2023-08-26T13:37:32.004 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ mkfname cluster2-mirror-test.status 2023-08-26T13:37:32.004 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ echo cluster2-mirror-test.status 2023-08-26T13:37:32.004 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ sed -e 's|[/ ]|_|g' 2023-08-26T13:37:32.005 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local status_log=/tmp/tmp.TfDn6BcQ3N/cluster2-mirror-test.status 2023-08-26T13:37:32.005 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ mkfname cluster1-mirror-test.status 2023-08-26T13:37:32.005 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ echo cluster1-mirror-test.status 2023-08-26T13:37:32.005 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ sed -e 's|[/ ]|_|g' 2023-08-26T13:37:32.006 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local local_status_log=/tmp/tmp.TfDn6BcQ3N/cluster1-mirror-test.status 2023-08-26T13:37:32.006 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ mirror_image_snapshot cluster2 mirror test 2023-08-26T13:37:32.006 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster2 2023-08-26T13:37:32.007 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T13:37:32.007 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=test 2023-08-26T13:37:32.007 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/test 2023-08-26T13:37:33.628 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:Snapshot ID: 21 2023-08-26T13:37:33.646 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ get_newest_mirror_snapshot cluster2 mirror test /tmp/tmp.TfDn6BcQ3N/cluster2-mirror-test.status 2023-08-26T13:37:33.646 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster2 2023-08-26T13:37:33.647 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T13:37:33.647 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=test 2023-08-26T13:37:33.647 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local log=/tmp/tmp.TfDn6BcQ3N/cluster2-mirror-test.status 2023-08-26T13:37:33.647 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rbd --cluster cluster2 snap list --all mirror/test --format xml 2023-08-26T13:37:33.648 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-08-26T13:37:33.709 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ xmlstarlet sel -t -v //snapshot/id 2023-08-26T13:37:33.711 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local snapshot_id=21 2023-08-26T13:37:33.712 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ true 2023-08-26T13:37:33.712 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-08-26T13:37:33.712 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ sleep 0.2 2023-08-26T13:37:33.913 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.TfDn6BcQ3N/cluster1-mirror-test.status 2023-08-26T13:37:33.913 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster1 2023-08-26T13:37:33.914 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T13:37:33.914 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=test 2023-08-26T13:37:33.914 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local log=/tmp/tmp.TfDn6BcQ3N/cluster1-mirror-test.status 2023-08-26T13:37:33.914 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2023-08-26T13:37:33.915 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-08-26T13:37:33.961 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ true 2023-08-26T13:37:33.961 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-08-26T13:37:33.962 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:-:1.1: Document is empty 2023-08-26T13:37:33.962 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr: 2023-08-26T13:37:33.963 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:^ 2023-08-26T13:37:33.963 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local primary_snapshot_id= 2023-08-26T13:37:33.963 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ test 21 = '' 2023-08-26T13:37:33.964 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-08-26T13:37:33.964 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ sleep 0.4 2023-08-26T13:37:34.364 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.TfDn6BcQ3N/cluster1-mirror-test.status 2023-08-26T13:37:34.364 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster1 2023-08-26T13:37:34.365 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T13:37:34.365 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=test 2023-08-26T13:37:34.365 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local log=/tmp/tmp.TfDn6BcQ3N/cluster1-mirror-test.status 2023-08-26T13:37:34.365 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2023-08-26T13:37:34.365 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-08-26T13:37:34.412 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ true 2023-08-26T13:37:34.412 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-08-26T13:37:34.413 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:-:1.1: Document is empty 2023-08-26T13:37:34.413 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr: 2023-08-26T13:37:34.413 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:^ 2023-08-26T13:37:34.414 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local primary_snapshot_id= 2023-08-26T13:37:34.414 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ test 21 = '' 2023-08-26T13:37:34.415 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-08-26T13:37:34.415 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ sleep 0.8 2023-08-26T13:37:35.215 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.TfDn6BcQ3N/cluster1-mirror-test.status 2023-08-26T13:37:35.216 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster1 2023-08-26T13:37:35.216 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T13:37:35.216 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=test 2023-08-26T13:37:35.217 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local log=/tmp/tmp.TfDn6BcQ3N/cluster1-mirror-test.status 2023-08-26T13:37:35.217 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2023-08-26T13:37:35.217 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-08-26T13:37:35.294 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ true 2023-08-26T13:37:35.294 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-08-26T13:37:35.295 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:-:1.1: Document is empty 2023-08-26T13:37:35.295 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr: 2023-08-26T13:37:35.296 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:^ 2023-08-26T13:37:35.296 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local primary_snapshot_id= 2023-08-26T13:37:35.296 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ test 21 = '' 2023-08-26T13:37:35.297 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-08-26T13:37:35.297 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ sleep 1.6 2023-08-26T13:37:36.897 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.TfDn6BcQ3N/cluster1-mirror-test.status 2023-08-26T13:37:36.897 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster1 2023-08-26T13:37:36.898 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T13:37:36.898 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=test 2023-08-26T13:37:36.898 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local log=/tmp/tmp.TfDn6BcQ3N/cluster1-mirror-test.status 2023-08-26T13:37:36.899 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2023-08-26T13:37:36.899 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-08-26T13:37:36.945 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-08-26T13:37:36.947 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local primary_snapshot_id=21 2023-08-26T13:37:36.947 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ test 21 = 21 2023-08-26T13:37:36.947 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ return 0 2023-08-26T13:37:36.948 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ wait_for_snap_present cluster1 mirror test snap5 2023-08-26T13:37:36.948 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster1 2023-08-26T13:37:36.948 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T13:37:36.949 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=test 2023-08-26T13:37:36.949 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local snap_name=snap5 2023-08-26T13:37:36.949 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local s 2023-08-26T13:37:36.949 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2023-08-26T13:37:36.950 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ sleep 1 2023-08-26T13:37:37.948 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rbd --cluster cluster1 info mirror/test@snap5 2023-08-26T13:37:37.996 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:rbd image 'test': 2023-08-26T13:37:37.996 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: size 512 MiB in 128 objects 2023-08-26T13:37:37.996 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: order 22 (4 MiB objects) 2023-08-26T13:37:37.997 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: snapshot_count: 10 2023-08-26T13:37:37.997 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: id: 10ab632cdafc 2023-08-26T13:37:37.997 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: block_name_prefix: rbd_data.10ab632cdafc 2023-08-26T13:37:37.997 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: format: 2 2023-08-26T13:37:37.998 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: features: layering, exclusive-lock, non-primary 2023-08-26T13:37:37.998 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: op_features: 2023-08-26T13:37:37.998 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: flags: 2023-08-26T13:37:37.998 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: create_timestamp: Sat Aug 26 13:33:17 2023 2023-08-26T13:37:37.999 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: access_timestamp: Sat Aug 26 13:35:55 2023 2023-08-26T13:37:37.999 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: modify_timestamp: Sat Aug 26 13:33:17 2023 2023-08-26T13:37:37.999 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: protected: False 2023-08-26T13:37:37.999 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: mirroring state: enabled 2023-08-26T13:37:37.999 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: mirroring mode: snapshot 2023-08-26T13:37:38.000 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: mirroring global id: e2f2554d-659e-4467-b7f8-dddd7ccb0ee9 2023-08-26T13:37:38.000 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: mirroring primary: false 2023-08-26T13:37:38.001 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ return 0 2023-08-26T13:37:38.001 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ '[' -n snap4-clean ']' 2023-08-26T13:37:38.002 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ compare_image_snaps mirror test snap4-clean 2023-08-26T13:37:38.002 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T13:37:38.002 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=test 2023-08-26T13:37:38.002 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local snap_name=snap4-clean 2023-08-26T13:37:38.002 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local ret=0 2023-08-26T13:37:38.002 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local rmt_export=/tmp/tmp.TfDn6BcQ3N/cluster2-mirror-test.export 2023-08-26T13:37:38.003 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local loc_export=/tmp/tmp.TfDn6BcQ3N/cluster1-mirror-test.export 2023-08-26T13:37:38.003 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rm -f /tmp/tmp.TfDn6BcQ3N/cluster2-mirror-test.export /tmp/tmp.TfDn6BcQ3N/cluster1-mirror-test.export 2023-08-26T13:37:38.003 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rbd --cluster cluster2 -p mirror export test@snap4-clean /tmp/tmp.TfDn6BcQ3N/cluster2-mirror-test.export 2023-08-26T13:37:38.582 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2023-08-26T13:37:38.592 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rbd --cluster cluster1 -p mirror export test@snap4-clean /tmp/tmp.TfDn6BcQ3N/cluster1-mirror-test.export 2023-08-26T13:37:39.426 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2023-08-26T13:37:39.434 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ cmp /tmp/tmp.TfDn6BcQ3N/cluster2-mirror-test.export /tmp/tmp.TfDn6BcQ3N/cluster1-mirror-test.export 2023-08-26T13:37:39.650 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rm -f /tmp/tmp.TfDn6BcQ3N/cluster2-mirror-test.export /tmp/tmp.TfDn6BcQ3N/cluster1-mirror-test.export 2023-08-26T13:37:39.794 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ return 0 2023-08-26T13:37:39.794 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ compare_image_snaps mirror test snap5 2023-08-26T13:37:39.794 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T13:37:39.795 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=test 2023-08-26T13:37:39.795 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local snap_name=snap5 2023-08-26T13:37:39.795 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local ret=0 2023-08-26T13:37:39.795 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local rmt_export=/tmp/tmp.TfDn6BcQ3N/cluster2-mirror-test.export 2023-08-26T13:37:39.795 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local loc_export=/tmp/tmp.TfDn6BcQ3N/cluster1-mirror-test.export 2023-08-26T13:37:39.796 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rm -f /tmp/tmp.TfDn6BcQ3N/cluster2-mirror-test.export /tmp/tmp.TfDn6BcQ3N/cluster1-mirror-test.export 2023-08-26T13:37:39.796 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rbd --cluster cluster2 -p mirror export test@snap5 /tmp/tmp.TfDn6BcQ3N/cluster2-mirror-test.export 2023-08-26T13:37:40.387 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2023-08-26T13:37:40.397 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rbd --cluster cluster1 -p mirror export test@snap5 /tmp/tmp.TfDn6BcQ3N/cluster1-mirror-test.export 2023-08-26T13:37:41.129 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2023-08-26T13:37:41.138 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ cmp /tmp/tmp.TfDn6BcQ3N/cluster2-mirror-test.export /tmp/tmp.TfDn6BcQ3N/cluster1-mirror-test.export 2023-08-26T13:37:41.352 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rm -f /tmp/tmp.TfDn6BcQ3N/cluster2-mirror-test.export /tmp/tmp.TfDn6BcQ3N/cluster1-mirror-test.export 2023-08-26T13:37:41.495 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ return 0 2023-08-26T13:37:41.496 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ clean_snap_name=snap5-clean 2023-08-26T13:37:41.496 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ create_snap cluster2 mirror test snap5-clean 2023-08-26T13:37:41.496 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster2 2023-08-26T13:37:41.496 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T13:37:41.497 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=test 2023-08-26T13:37:41.497 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local snap_name=snap5-clean 2023-08-26T13:37:41.497 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rbd --cluster cluster2 -p mirror snap create test@snap5-clean --debug-rbd=20 --debug-journaler=20 2023-08-26T13:37:41.587 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ for i in `seq 1 10` 2023-08-26T13:37:41.587 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ stress_write_image cluster2 mirror test 2023-08-26T13:37:41.588 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster2 2023-08-26T13:37:41.588 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T13:37:41.588 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=test 2023-08-26T13:37:41.588 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2023-08-26T13:37:41.589 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local duration=6 2023-08-26T13:37:41.589 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ set +e 2023-08-26T13:37:41.589 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ timeout 6s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror test --debug-rbd=20 --debug-journaler=20 2023-08-26T13:37:41.747 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:100 IOs, 610304 bytes 2023-08-26T13:37:41.863 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:200 IOs, 13516800 bytes 2023-08-26T13:37:41.979 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:300 IOs, 30215168 bytes 2023-08-26T13:37:42.070 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:400 IOs, 41375744 bytes 2023-08-26T13:37:42.261 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:500 IOs, 102474752 bytes 2023-08-26T13:37:42.368 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:600 IOs, 124590080 bytes 2023-08-26T13:37:42.601 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:700 IOs, 197301248 bytes 2023-08-26T13:37:42.786 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:800 IOs, 254541824 bytes 2023-08-26T13:37:42.940 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:900 IOs, 295697408 bytes 2023-08-26T13:37:43.011 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:1000 IOs, 296314880 bytes 2023-08-26T13:37:43.194 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:1100 IOs, 367598592 bytes 2023-08-26T13:37:43.346 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:1200 IOs, 401236992 bytes 2023-08-26T13:37:43.503 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:1300 IOs, 450799616 bytes 2023-08-26T13:37:43.694 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:1400 IOs, 488804352 bytes 2023-08-26T13:37:43.821 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:1500 IOs, 523054080 bytes 2023-08-26T13:37:43.962 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:1600 IOs, 555342848 bytes 2023-08-26T13:37:44.066 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:1700 IOs, 569565184 bytes 2023-08-26T13:37:44.229 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:1800 IOs, 623245312 bytes 2023-08-26T13:37:44.390 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:1900 IOs, 653240320 bytes 2023-08-26T13:37:44.465 INFO:tasks.rbd_mirror_thrash:kill cluster1.client.mirror.0 2023-08-26T13:37:44.466 INFO:tasks.rbd_mirror.cluster1.client.mirror.0:Sent signal 15 2023-08-26T13:37:44.466 INFO:tasks.rbd_mirror_thrash:waiting for 2 secs before reviving daemons 2023-08-26T13:37:44.467 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi124.stderr:2023-08-26T13:37:44.465+0000 7f25a7964700 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.0 (PID: 109470) UID: 1000 2023-08-26T13:37:44.596 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:2000 IOs, 715603968 bytes 2023-08-26T13:37:44.689 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:2100 IOs, 724960256 bytes 2023-08-26T13:37:44.770 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:2200 IOs, 732921856 bytes 2023-08-26T13:37:44.960 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:2300 IOs, 791477248 bytes 2023-08-26T13:37:45.177 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:2400 IOs, 866211840 bytes 2023-08-26T13:37:45.285 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:2500 IOs, 883614720 bytes 2023-08-26T13:37:45.414 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:2600 IOs, 913879040 bytes 2023-08-26T13:37:45.654 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:2700 IOs, 1010232320 bytes 2023-08-26T13:37:45.800 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:2800 IOs, 1042138112 bytes 2023-08-26T13:37:45.959 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:2900 IOs, 1123481600 bytes 2023-08-26T13:37:46.191 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:3000 IOs, 1204012032 bytes 2023-08-26T13:37:46.376 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:3100 IOs, 1248168960 bytes 2023-08-26T13:37:46.467 INFO:tasks.rbd_mirror_thrash:waiting for cluster1.client.mirror.0 2023-08-26T13:37:46.467 INFO:teuthology.orchestra.run:waiting for 600 2023-08-26T13:37:46.467 INFO:tasks.rbd_mirror_thrash:reviving cluster1.client.mirror.0 2023-08-26T13:37:46.468 INFO:tasks.rbd_mirror.cluster1.client.mirror.0:Restarting daemon 2023-08-26T13:37:46.468 DEBUG:teuthology.orchestra.run.smithi124:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.0 2023-08-26T13:37:46.471 INFO:tasks.rbd_mirror.cluster1.client.mirror.0:Started 2023-08-26T13:37:46.471 INFO:tasks.rbd_mirror_thrash:waiting for 72 secs before thrashing 2023-08-26T13:37:46.558 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:3200 IOs, 1308390400 bytes 2023-08-26T13:37:46.761 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:3300 IOs, 1356246016 bytes 2023-08-26T13:37:46.962 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:3400 IOs, 1426100224 bytes 2023-08-26T13:37:47.113 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:3500 IOs, 1453010944 bytes 2023-08-26T13:37:47.326 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:3600 IOs, 1523577856 bytes 2023-08-26T13:37:47.465 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:3700 IOs, 1550441472 bytes 2023-08-26T13:37:47.593 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ error_code=124 2023-08-26T13:37:47.593 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ set -e 2023-08-26T13:37:47.594 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ '[' 124 -eq 124 ']' 2023-08-26T13:37:47.594 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ return 0 2023-08-26T13:37:47.594 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ wait_for_status_in_pool_dir cluster1 mirror test up+replaying 2023-08-26T13:37:47.594 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster1 2023-08-26T13:37:47.595 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T13:37:47.595 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=test 2023-08-26T13:37:47.595 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local state_pattern=up+replaying 2023-08-26T13:37:47.595 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local description_pattern= 2023-08-26T13:37:47.596 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local service_pattern= 2023-08-26T13:37:47.596 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-08-26T13:37:47.596 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ sleep 1 2023-08-26T13:37:48.594 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ test_status_in_pool_dir cluster1 mirror test up+replaying '' '' 2023-08-26T13:37:48.595 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster1 2023-08-26T13:37:48.595 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T13:37:48.595 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=test 2023-08-26T13:37:48.595 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local state_pattern=up+replaying 2023-08-26T13:37:48.596 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local description_pattern= 2023-08-26T13:37:48.596 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local service_pattern= 2023-08-26T13:37:48.596 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ mkfname cluster1-mirror-test.mirror_status 2023-08-26T13:37:48.596 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ echo cluster1-mirror-test.mirror_status 2023-08-26T13:37:48.596 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ sed -e 's|[/ ]|_|g' 2023-08-26T13:37:48.597 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local status_log=/tmp/tmp.TfDn6BcQ3N/cluster1-mirror-test.mirror_status 2023-08-26T13:37:48.597 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ CEPH_ARGS= 2023-08-26T13:37:48.597 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rbd --cluster cluster1 mirror image status mirror/test 2023-08-26T13:37:48.598 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ tee /tmp/tmp.TfDn6BcQ3N/cluster1-mirror-test.mirror_status 2023-08-26T13:37:48.649 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:test: 2023-08-26T13:37:48.649 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr: global_id: e2f2554d-659e-4467-b7f8-dddd7ccb0ee9 2023-08-26T13:37:48.649 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr: state: up+replaying 2023-08-26T13:37:48.650 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr: description: replaying, {"bytes_per_second":17857024.0,"bytes_per_snapshot":533805568.0,"last_snapshot_bytes":535710720,"last_snapshot_sync_seconds":1,"local_snapshot_timestamp":1693057052,"remote_snapshot_timestamp":1693057052,"replay_state":"idle"} 2023-08-26T13:37:48.650 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr: service: mirror.2 on smithi124 2023-08-26T13:37:48.650 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr: last_update: 2023-08-26 13:37:45 2023-08-26T13:37:48.650 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr: peer_sites: 2023-08-26T13:37:48.650 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr: name: cluster2 2023-08-26T13:37:48.651 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr: state: up+stopped 2023-08-26T13:37:48.651 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr: description: local image is primary 2023-08-26T13:37:48.651 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr: last_update: 2023-08-26 13:37:45 2023-08-26T13:37:48.654 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ grep '^ state: .*up+replaying' /tmp/tmp.TfDn6BcQ3N/cluster1-mirror-test.mirror_status 2023-08-26T13:37:48.655 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: state: up+replaying 2023-08-26T13:37:48.655 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ grep '^ description: .*' /tmp/tmp.TfDn6BcQ3N/cluster1-mirror-test.mirror_status 2023-08-26T13:37:48.656 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: description: replaying, {"bytes_per_second":17857024.0,"bytes_per_snapshot":533805568.0,"last_snapshot_bytes":535710720,"last_snapshot_sync_seconds":1,"local_snapshot_timestamp":1693057052,"remote_snapshot_timestamp":1693057052,"replay_state":"idle"} 2023-08-26T13:37:48.656 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ '[' -n '' ']' 2023-08-26T13:37:48.657 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ echo up+replaying 2023-08-26T13:37:48.657 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ grep '^up+' 2023-08-26T13:37:48.657 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:up+replaying 2023-08-26T13:37:48.658 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ grep 'service: *mirror..* on ' /tmp/tmp.TfDn6BcQ3N/cluster1-mirror-test.mirror_status 2023-08-26T13:37:48.658 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: service: mirror.2 on smithi124 2023-08-26T13:37:48.659 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' /tmp/tmp.TfDn6BcQ3N/cluster1-mirror-test.mirror_status 2023-08-26T13:37:48.660 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local 'last_update=2023-08-26 13:37:45' 2023-08-26T13:37:48.660 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ test_mirror_pool_status_verbose cluster1 mirror test up+replaying '2023-08-26 13:37:45' 2023-08-26T13:37:48.660 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster1 2023-08-26T13:37:48.661 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T13:37:48.661 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=test 2023-08-26T13:37:48.661 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local state_pattern=up+replaying 2023-08-26T13:37:48.661 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local 'prev_last_update=2023-08-26 13:37:45' 2023-08-26T13:37:48.661 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ mkfname cluster1-mirror.mirror_status 2023-08-26T13:37:48.662 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ echo cluster1-mirror.mirror_status 2023-08-26T13:37:48.662 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ sed -e 's|[/ ]|_|g' 2023-08-26T13:37:48.662 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local status_log=/tmp/tmp.TfDn6BcQ3N/cluster1-mirror.mirror_status 2023-08-26T13:37:48.663 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2023-08-26T13:37:48.699 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:rbd: failed to query services: (13) Permission denied 2023-08-26T13:37:48.721 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local last_update state 2023-08-26T13:37:48.721 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update' 2023-08-26T13:37:48.723 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ last_update='2023-08-26 13:37:45' 2023-08-26T13:37:48.723 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state' 2023-08-26T13:37:48.725 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ state=up+replaying 2023-08-26T13:37:48.725 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ echo up+replaying 2023-08-26T13:37:48.725 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ grep up+replaying 2023-08-26T13:37:48.726 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:up+replaying 2023-08-26T13:37:48.727 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ return 0 2023-08-26T13:37:48.727 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ return 0 2023-08-26T13:37:48.727 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ snap_name=snap6 2023-08-26T13:37:48.728 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ create_snap cluster2 mirror test snap6 2023-08-26T13:37:48.728 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster2 2023-08-26T13:37:48.728 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T13:37:48.728 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=test 2023-08-26T13:37:48.728 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local snap_name=snap6 2023-08-26T13:37:48.729 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rbd --cluster cluster2 -p mirror snap create test@snap6 --debug-rbd=20 --debug-journaler=20 2023-08-26T13:38:18.624 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ wait_for_image_replay_started cluster1 mirror test 2023-08-26T13:38:18.625 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster1 2023-08-26T13:38:18.625 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T13:38:18.625 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=test 2023-08-26T13:38:18.625 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ wait_for_image_replay_state cluster1 mirror test started 2023-08-26T13:38:18.625 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster1 2023-08-26T13:38:18.626 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T13:38:18.626 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=test 2023-08-26T13:38:18.626 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local state=started 2023-08-26T13:38:18.626 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local s 2023-08-26T13:38:18.626 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-08-26T13:38:18.627 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ sleep 1 2023-08-26T13:38:19.461 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.0 with skip (0.49634403303003405) > weight (0.25) 2023-08-26T13:38:19.462 INFO:tasks.rbd_mirror_thrash:kill cluster2.client.mirror.1 2023-08-26T13:38:19.462 INFO:tasks.rbd_mirror.cluster2.client.mirror.1:Sent signal 15 2023-08-26T13:38:19.462 INFO:tasks.rbd_mirror_thrash:waiting for 2 secs before reviving daemons 2023-08-26T13:38:19.463 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi124.stderr:2023-08-26T13:38:19.462+0000 7fdbaae80700 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.1 (PID: 109988) UID: 1000 2023-08-26T13:38:19.626 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ test_image_replay_state cluster1 mirror test started 2023-08-26T13:38:19.626 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster1 2023-08-26T13:38:19.626 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T13:38:19.627 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=test 2023-08-26T13:38:19.627 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local test_state=started 2023-08-26T13:38:19.627 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local status_result 2023-08-26T13:38:19.627 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local current_state=stopped 2023-08-26T13:38:19.627 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ admin_daemons cluster1 rbd mirror status mirror/test 2023-08-26T13:38:19.628 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local cluster_instance=cluster1 2023-08-26T13:38:19.628 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ shift 2023-08-26T13:38:19.628 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ grep -i state 2023-08-26T13:38:19.628 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local cluster=cluster1 2023-08-26T13:38:19.629 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local instance=cluster1 2023-08-26T13:38:19.629 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local loop_instance 2023-08-26T13:38:19.629 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-08-26T13:38:19.630 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ sleep 0 2023-08-26T13:38:19.635 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ '[' cluster1 '!=' cluster1 ']' 2023-08-26T13:38:19.636 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ seq 0 3 2023-08-26T13:38:19.636 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-08-26T13:38:19.636 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test 2023-08-26T13:38:19.637 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local cluster=cluster1:0 2023-08-26T13:38:19.637 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ shift 2023-08-26T13:38:19.638 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local instance 2023-08-26T13:38:19.638 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ set_cluster_instance cluster1:0 cluster instance 2023-08-26T13:38:19.638 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local val=cluster1:0 2023-08-26T13:38:19.639 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local cluster_var_name=cluster 2023-08-26T13:38:19.639 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local instance_var_name=instance 2023-08-26T13:38:19.639 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ cluster=cluster1 2023-08-26T13:38:19.640 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ instance=0 2023-08-26T13:38:19.640 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ '[' 0 = cluster1:0 ']' 2023-08-26T13:38:19.640 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ eval cluster=cluster1 2023-08-26T13:38:19.641 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ cluster=cluster1 2023-08-26T13:38:19.641 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ eval instance=0 2023-08-26T13:38:19.642 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ instance=0 2023-08-26T13:38:19.642 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ daemon_asok_file cluster1:0 cluster1 2023-08-26T13:38:19.642 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ local local_cluster=cluster1:0 2023-08-26T13:38:19.643 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ local cluster=cluster1 2023-08-26T13:38:19.643 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ local instance 2023-08-26T13:38:19.643 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2023-08-26T13:38:19.644 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ local val=cluster1:0 2023-08-26T13:38:19.644 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ local cluster_var_name=local_cluster 2023-08-26T13:38:19.644 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ local instance_var_name=instance 2023-08-26T13:38:19.645 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ cluster=cluster1 2023-08-26T13:38:19.645 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ instance=0 2023-08-26T13:38:19.645 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ '[' 0 = cluster1:0 ']' 2023-08-26T13:38:19.646 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ eval local_cluster=cluster1 2023-08-26T13:38:19.646 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++++ local_cluster=cluster1 2023-08-26T13:38:19.646 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ eval instance=0 2023-08-26T13:38:19.647 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++++ instance=0 2023-08-26T13:38:19.647 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2023-08-26T13:38:19.650 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-08-26T13:38:19.651 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-08-26T13:38:19.651 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-08-26T13:38:19.651 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2023-08-26T13:38:19.749 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:Invalid command: unused arguments: ['mirror/test'] 2023-08-26T13:38:19.749 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:rbd mirror status : get status for rbd mirror 2023-08-26T13:38:19.750 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:admin_socket: invalid command 2023-08-26T13:38:19.752 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-08-26T13:38:19.752 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/test 2023-08-26T13:38:19.753 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local cluster=cluster1:1 2023-08-26T13:38:19.753 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ shift 2023-08-26T13:38:19.753 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local instance 2023-08-26T13:38:19.753 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ set_cluster_instance cluster1:1 cluster instance 2023-08-26T13:38:19.754 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local val=cluster1:1 2023-08-26T13:38:19.754 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local cluster_var_name=cluster 2023-08-26T13:38:19.754 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local instance_var_name=instance 2023-08-26T13:38:19.754 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ cluster=cluster1 2023-08-26T13:38:19.754 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ instance=1 2023-08-26T13:38:19.755 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ '[' 1 = cluster1:1 ']' 2023-08-26T13:38:19.755 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ eval cluster=cluster1 2023-08-26T13:38:19.755 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ cluster=cluster1 2023-08-26T13:38:19.755 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ eval instance=1 2023-08-26T13:38:19.756 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ instance=1 2023-08-26T13:38:19.756 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ daemon_asok_file cluster1:1 cluster1 2023-08-26T13:38:19.756 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ local local_cluster=cluster1:1 2023-08-26T13:38:19.757 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ local cluster=cluster1 2023-08-26T13:38:19.757 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ local instance 2023-08-26T13:38:19.757 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2023-08-26T13:38:19.758 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ local val=cluster1:1 2023-08-26T13:38:19.758 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ local cluster_var_name=local_cluster 2023-08-26T13:38:19.758 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ local instance_var_name=instance 2023-08-26T13:38:19.759 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ cluster=cluster1 2023-08-26T13:38:19.759 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ instance=1 2023-08-26T13:38:19.759 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ '[' 1 = cluster1:1 ']' 2023-08-26T13:38:19.760 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ eval local_cluster=cluster1 2023-08-26T13:38:19.760 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++++ local_cluster=cluster1 2023-08-26T13:38:19.760 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ eval instance=1 2023-08-26T13:38:19.761 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++++ instance=1 2023-08-26T13:38:19.761 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2023-08-26T13:38:19.771 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-08-26T13:38:19.771 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-08-26T13:38:19.771 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-08-26T13:38:19.771 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/test 2023-08-26T13:38:19.865 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:Invalid command: unused arguments: ['mirror/test'] 2023-08-26T13:38:19.865 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:rbd mirror status : get status for rbd mirror 2023-08-26T13:38:19.865 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:admin_socket: invalid command 2023-08-26T13:38:19.868 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-08-26T13:38:19.868 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/test 2023-08-26T13:38:19.868 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local cluster=cluster1:2 2023-08-26T13:38:19.868 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ shift 2023-08-26T13:38:19.869 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local instance 2023-08-26T13:38:19.869 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ set_cluster_instance cluster1:2 cluster instance 2023-08-26T13:38:19.869 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local val=cluster1:2 2023-08-26T13:38:19.869 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local cluster_var_name=cluster 2023-08-26T13:38:19.869 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local instance_var_name=instance 2023-08-26T13:38:19.870 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ cluster=cluster1 2023-08-26T13:38:19.870 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ instance=2 2023-08-26T13:38:19.870 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ '[' 2 = cluster1:2 ']' 2023-08-26T13:38:19.870 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ eval cluster=cluster1 2023-08-26T13:38:19.870 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ cluster=cluster1 2023-08-26T13:38:19.870 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ eval instance=2 2023-08-26T13:38:19.871 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ instance=2 2023-08-26T13:38:19.871 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ daemon_asok_file cluster1:2 cluster1 2023-08-26T13:38:19.871 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ local local_cluster=cluster1:2 2023-08-26T13:38:19.871 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ local cluster=cluster1 2023-08-26T13:38:19.871 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ local instance 2023-08-26T13:38:19.872 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2023-08-26T13:38:19.872 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ local val=cluster1:2 2023-08-26T13:38:19.872 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ local cluster_var_name=local_cluster 2023-08-26T13:38:19.872 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ local instance_var_name=instance 2023-08-26T13:38:19.872 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ cluster=cluster1 2023-08-26T13:38:19.873 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ instance=2 2023-08-26T13:38:19.873 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ '[' 2 = cluster1:2 ']' 2023-08-26T13:38:19.873 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ eval local_cluster=cluster1 2023-08-26T13:38:19.873 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++++ local_cluster=cluster1 2023-08-26T13:38:19.873 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ eval instance=2 2023-08-26T13:38:19.873 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++++ instance=2 2023-08-26T13:38:19.874 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2023-08-26T13:38:19.887 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-08-26T13:38:19.887 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-08-26T13:38:19.887 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-08-26T13:38:19.887 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/test 2023-08-26T13:38:19.992 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ return 0 2023-08-26T13:38:19.992 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ status_result=' "state": "Replaying"' 2023-08-26T13:38:19.993 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ echo ' "state": "Replaying"' 2023-08-26T13:38:19.993 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ grep -i Replaying 2023-08-26T13:38:19.993 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: "state": "Replaying" 2023-08-26T13:38:19.995 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ current_state=started 2023-08-26T13:38:19.996 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ test started = started 2023-08-26T13:38:19.996 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ return 0 2023-08-26T13:38:19.996 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror test 2023-08-26T13:38:19.996 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local local_cluster=cluster1 2023-08-26T13:38:19.996 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster2 2023-08-26T13:38:19.997 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T13:38:19.997 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=test 2023-08-26T13:38:19.997 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ '[' snapshot = journal ']' 2023-08-26T13:38:19.997 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ '[' snapshot = snapshot ']' 2023-08-26T13:38:19.997 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror test 2023-08-26T13:38:19.998 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local local_cluster=cluster1 2023-08-26T13:38:19.998 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster2 2023-08-26T13:38:19.998 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T13:38:19.998 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=test 2023-08-26T13:38:19.998 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ mkfname cluster2-mirror-test.status 2023-08-26T13:38:19.998 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ echo cluster2-mirror-test.status 2023-08-26T13:38:19.999 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ sed -e 's|[/ ]|_|g' 2023-08-26T13:38:19.999 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local status_log=/tmp/tmp.TfDn6BcQ3N/cluster2-mirror-test.status 2023-08-26T13:38:19.999 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ mkfname cluster1-mirror-test.status 2023-08-26T13:38:19.999 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ echo cluster1-mirror-test.status 2023-08-26T13:38:19.999 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ sed -e 's|[/ ]|_|g' 2023-08-26T13:38:20.000 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local local_status_log=/tmp/tmp.TfDn6BcQ3N/cluster1-mirror-test.status 2023-08-26T13:38:20.000 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ mirror_image_snapshot cluster2 mirror test 2023-08-26T13:38:20.001 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster2 2023-08-26T13:38:20.001 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T13:38:20.001 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=test 2023-08-26T13:38:20.001 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/test 2023-08-26T13:38:21.464 INFO:tasks.rbd_mirror_thrash:waiting for cluster2.client.mirror.1 2023-08-26T13:38:21.464 INFO:teuthology.orchestra.run:waiting for 600 2023-08-26T13:38:21.464 INFO:tasks.rbd_mirror_thrash:reviving cluster2.client.mirror.1 2023-08-26T13:38:21.465 INFO:tasks.rbd_mirror.cluster2.client.mirror.1:Restarting daemon 2023-08-26T13:38:21.465 DEBUG:teuthology.orchestra.run.smithi124:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.1 2023-08-26T13:38:21.468 INFO:tasks.rbd_mirror.cluster2.client.mirror.1:Started 2023-08-26T13:38:21.468 INFO:tasks.rbd_mirror_thrash:waiting for 82 secs before thrashing 2023-08-26T13:38:21.631 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:Snapshot ID: 25 2023-08-26T13:38:21.648 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ get_newest_mirror_snapshot cluster2 mirror test /tmp/tmp.TfDn6BcQ3N/cluster2-mirror-test.status 2023-08-26T13:38:21.648 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster2 2023-08-26T13:38:21.648 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T13:38:21.648 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=test 2023-08-26T13:38:21.649 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local log=/tmp/tmp.TfDn6BcQ3N/cluster2-mirror-test.status 2023-08-26T13:38:21.649 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rbd --cluster cluster2 snap list --all mirror/test --format xml 2023-08-26T13:38:21.649 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-08-26T13:38:21.722 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ xmlstarlet sel -t -v //snapshot/id 2023-08-26T13:38:21.725 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local snapshot_id=25 2023-08-26T13:38:21.725 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ true 2023-08-26T13:38:21.725 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-08-26T13:38:21.726 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ sleep 0.2 2023-08-26T13:38:21.927 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.TfDn6BcQ3N/cluster1-mirror-test.status 2023-08-26T13:38:21.927 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster1 2023-08-26T13:38:21.927 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T13:38:21.928 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=test 2023-08-26T13:38:21.928 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local log=/tmp/tmp.TfDn6BcQ3N/cluster1-mirror-test.status 2023-08-26T13:38:21.928 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2023-08-26T13:38:21.928 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-08-26T13:38:21.974 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ true 2023-08-26T13:38:21.975 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-08-26T13:38:21.975 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:-:1.1: Document is empty 2023-08-26T13:38:21.976 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr: 2023-08-26T13:38:21.976 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:^ 2023-08-26T13:38:21.977 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local primary_snapshot_id= 2023-08-26T13:38:21.977 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ test 25 = '' 2023-08-26T13:38:21.977 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-08-26T13:38:21.977 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ sleep 0.4 2023-08-26T13:38:22.377 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.TfDn6BcQ3N/cluster1-mirror-test.status 2023-08-26T13:38:22.378 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster1 2023-08-26T13:38:22.378 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T13:38:22.378 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=test 2023-08-26T13:38:22.378 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local log=/tmp/tmp.TfDn6BcQ3N/cluster1-mirror-test.status 2023-08-26T13:38:22.378 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2023-08-26T13:38:22.379 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-08-26T13:38:22.425 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ true 2023-08-26T13:38:22.426 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-08-26T13:38:22.427 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:-:1.1: Document is empty 2023-08-26T13:38:22.427 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr: 2023-08-26T13:38:22.427 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:^ 2023-08-26T13:38:22.428 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local primary_snapshot_id= 2023-08-26T13:38:22.428 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ test 25 = '' 2023-08-26T13:38:22.428 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-08-26T13:38:22.429 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ sleep 0.8 2023-08-26T13:38:23.229 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.TfDn6BcQ3N/cluster1-mirror-test.status 2023-08-26T13:38:23.229 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster1 2023-08-26T13:38:23.229 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T13:38:23.230 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=test 2023-08-26T13:38:23.230 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local log=/tmp/tmp.TfDn6BcQ3N/cluster1-mirror-test.status 2023-08-26T13:38:23.230 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2023-08-26T13:38:23.230 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-08-26T13:38:23.338 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ true 2023-08-26T13:38:23.339 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-08-26T13:38:23.341 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:-:1.1: Document is empty 2023-08-26T13:38:23.341 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr: 2023-08-26T13:38:23.341 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:^ 2023-08-26T13:38:23.342 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local primary_snapshot_id= 2023-08-26T13:38:23.342 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ test 25 = '' 2023-08-26T13:38:23.342 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-08-26T13:38:23.343 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ sleep 1.6 2023-08-26T13:38:24.944 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.TfDn6BcQ3N/cluster1-mirror-test.status 2023-08-26T13:38:24.944 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster1 2023-08-26T13:38:24.944 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T13:38:24.945 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=test 2023-08-26T13:38:24.945 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local log=/tmp/tmp.TfDn6BcQ3N/cluster1-mirror-test.status 2023-08-26T13:38:24.945 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2023-08-26T13:38:24.945 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-08-26T13:38:24.995 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-08-26T13:38:24.998 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local primary_snapshot_id=25 2023-08-26T13:38:24.998 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ test 25 = 25 2023-08-26T13:38:24.998 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ return 0 2023-08-26T13:38:24.998 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ wait_for_snap_present cluster1 mirror test snap6 2023-08-26T13:38:24.999 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster1 2023-08-26T13:38:24.999 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T13:38:24.999 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=test 2023-08-26T13:38:24.999 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local snap_name=snap6 2023-08-26T13:38:24.999 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local s 2023-08-26T13:38:25.000 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2023-08-26T13:38:25.000 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ sleep 1 2023-08-26T13:38:25.999 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rbd --cluster cluster1 info mirror/test@snap6 2023-08-26T13:38:26.045 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:rbd image 'test': 2023-08-26T13:38:26.045 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: size 512 MiB in 128 objects 2023-08-26T13:38:26.046 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: order 22 (4 MiB objects) 2023-08-26T13:38:26.046 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: snapshot_count: 12 2023-08-26T13:38:26.046 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: id: 10ab632cdafc 2023-08-26T13:38:26.047 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: block_name_prefix: rbd_data.10ab632cdafc 2023-08-26T13:38:26.047 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: format: 2 2023-08-26T13:38:26.048 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: features: layering, exclusive-lock, non-primary 2023-08-26T13:38:26.048 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: op_features: 2023-08-26T13:38:26.048 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: flags: 2023-08-26T13:38:26.049 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: create_timestamp: Sat Aug 26 13:33:17 2023 2023-08-26T13:38:26.049 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: access_timestamp: Sat Aug 26 13:37:38 2023 2023-08-26T13:38:26.049 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: modify_timestamp: Sat Aug 26 13:33:17 2023 2023-08-26T13:38:26.050 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: protected: False 2023-08-26T13:38:26.050 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: mirroring state: enabled 2023-08-26T13:38:26.050 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: mirroring mode: snapshot 2023-08-26T13:38:26.051 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: mirroring global id: e2f2554d-659e-4467-b7f8-dddd7ccb0ee9 2023-08-26T13:38:26.051 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: mirroring primary: false 2023-08-26T13:38:26.053 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ return 0 2023-08-26T13:38:26.053 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ '[' -n snap5-clean ']' 2023-08-26T13:38:26.054 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ compare_image_snaps mirror test snap5-clean 2023-08-26T13:38:26.054 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T13:38:26.054 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=test 2023-08-26T13:38:26.055 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local snap_name=snap5-clean 2023-08-26T13:38:26.055 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local ret=0 2023-08-26T13:38:26.055 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local rmt_export=/tmp/tmp.TfDn6BcQ3N/cluster2-mirror-test.export 2023-08-26T13:38:26.056 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local loc_export=/tmp/tmp.TfDn6BcQ3N/cluster1-mirror-test.export 2023-08-26T13:38:26.056 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rm -f /tmp/tmp.TfDn6BcQ3N/cluster2-mirror-test.export /tmp/tmp.TfDn6BcQ3N/cluster1-mirror-test.export 2023-08-26T13:38:26.056 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rbd --cluster cluster2 -p mirror export test@snap5-clean /tmp/tmp.TfDn6BcQ3N/cluster2-mirror-test.export 2023-08-26T13:38:26.655 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2023-08-26T13:38:26.666 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rbd --cluster cluster1 -p mirror export test@snap5-clean /tmp/tmp.TfDn6BcQ3N/cluster1-mirror-test.export 2023-08-26T13:38:27.323 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2023-08-26T13:38:27.331 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ cmp /tmp/tmp.TfDn6BcQ3N/cluster2-mirror-test.export /tmp/tmp.TfDn6BcQ3N/cluster1-mirror-test.export 2023-08-26T13:38:27.544 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rm -f /tmp/tmp.TfDn6BcQ3N/cluster2-mirror-test.export /tmp/tmp.TfDn6BcQ3N/cluster1-mirror-test.export 2023-08-26T13:38:27.689 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ return 0 2023-08-26T13:38:27.690 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ compare_image_snaps mirror test snap6 2023-08-26T13:38:27.690 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T13:38:27.690 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=test 2023-08-26T13:38:27.690 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local snap_name=snap6 2023-08-26T13:38:27.690 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local ret=0 2023-08-26T13:38:27.690 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local rmt_export=/tmp/tmp.TfDn6BcQ3N/cluster2-mirror-test.export 2023-08-26T13:38:27.691 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local loc_export=/tmp/tmp.TfDn6BcQ3N/cluster1-mirror-test.export 2023-08-26T13:38:27.691 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rm -f /tmp/tmp.TfDn6BcQ3N/cluster2-mirror-test.export /tmp/tmp.TfDn6BcQ3N/cluster1-mirror-test.export 2023-08-26T13:38:27.691 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rbd --cluster cluster2 -p mirror export test@snap6 /tmp/tmp.TfDn6BcQ3N/cluster2-mirror-test.export 2023-08-26T13:38:28.270 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2023-08-26T13:38:28.280 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rbd --cluster cluster1 -p mirror export test@snap6 /tmp/tmp.TfDn6BcQ3N/cluster1-mirror-test.export 2023-08-26T13:38:29.050 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2023-08-26T13:38:29.060 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ cmp /tmp/tmp.TfDn6BcQ3N/cluster2-mirror-test.export /tmp/tmp.TfDn6BcQ3N/cluster1-mirror-test.export 2023-08-26T13:38:29.272 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rm -f /tmp/tmp.TfDn6BcQ3N/cluster2-mirror-test.export /tmp/tmp.TfDn6BcQ3N/cluster1-mirror-test.export 2023-08-26T13:38:29.414 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ return 0 2023-08-26T13:38:29.414 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ clean_snap_name=snap6-clean 2023-08-26T13:38:29.414 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ create_snap cluster2 mirror test snap6-clean 2023-08-26T13:38:29.414 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster2 2023-08-26T13:38:29.415 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T13:38:29.415 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=test 2023-08-26T13:38:29.415 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local snap_name=snap6-clean 2023-08-26T13:38:29.415 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rbd --cluster cluster2 -p mirror snap create test@snap6-clean --debug-rbd=20 --debug-journaler=20 2023-08-26T13:38:29.643 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ for i in `seq 1 10` 2023-08-26T13:38:29.643 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ stress_write_image cluster2 mirror test 2023-08-26T13:38:29.643 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster2 2023-08-26T13:38:29.643 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T13:38:29.643 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=test 2023-08-26T13:38:29.644 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2023-08-26T13:38:29.644 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local duration=7 2023-08-26T13:38:29.644 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ set +e 2023-08-26T13:38:29.645 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ timeout 7s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror test --debug-rbd=20 --debug-journaler=20 2023-08-26T13:38:29.844 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:100 IOs, 17132544 bytes 2023-08-26T13:38:29.949 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:200 IOs, 26394624 bytes 2023-08-26T13:38:30.155 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:300 IOs, 93130752 bytes 2023-08-26T13:38:30.277 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:400 IOs, 122372096 bytes 2023-08-26T13:38:30.416 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:500 IOs, 156513280 bytes 2023-08-26T13:38:30.736 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:600 IOs, 282193920 bytes 2023-08-26T13:38:30.970 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:700 IOs, 374657024 bytes 2023-08-26T13:38:31.166 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:800 IOs, 417809408 bytes 2023-08-26T13:38:31.292 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:900 IOs, 479827968 bytes 2023-08-26T13:38:31.574 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:1000 IOs, 550761472 bytes 2023-08-26T13:38:31.669 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:1100 IOs, 564979712 bytes 2023-08-26T13:38:31.816 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:1200 IOs, 607947776 bytes 2023-08-26T13:38:31.985 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:1300 IOs, 643871744 bytes 2023-08-26T13:38:32.108 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:1400 IOs, 669788160 bytes 2023-08-26T13:38:32.362 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:1500 IOs, 755763200 bytes 2023-08-26T13:38:32.535 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:1600 IOs, 808427520 bytes 2023-08-26T13:38:32.776 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:1700 IOs, 900184064 bytes 2023-08-26T13:38:33.054 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:1800 IOs, 1000112128 bytes 2023-08-26T13:38:33.214 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:1900 IOs, 1044384768 bytes 2023-08-26T13:38:33.318 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:2000 IOs, 1063734272 bytes 2023-08-26T13:38:33.497 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:2100 IOs, 1110359040 bytes 2023-08-26T13:38:33.694 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:2200 IOs, 1171095552 bytes 2023-08-26T13:38:33.904 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:2300 IOs, 1242425344 bytes 2023-08-26T13:38:34.041 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:2400 IOs, 1271434240 bytes 2023-08-26T13:38:34.137 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:2500 IOs, 1292733440 bytes 2023-08-26T13:38:34.298 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:2600 IOs, 1319687168 bytes 2023-08-26T13:38:34.499 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:2700 IOs, 1398448128 bytes 2023-08-26T13:38:34.712 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:2800 IOs, 1456061440 bytes 2023-08-26T13:38:34.799 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:2900 IOs, 1468324864 bytes 2023-08-26T13:38:35.066 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:3000 IOs, 1555008512 bytes 2023-08-26T13:38:35.235 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:3100 IOs, 1606669312 bytes 2023-08-26T13:38:35.397 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:3200 IOs, 1650174976 bytes 2023-08-26T13:38:35.493 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:3300 IOs, 1667367936 bytes 2023-08-26T13:38:35.677 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:3400 IOs, 1709016064 bytes 2023-08-26T13:38:35.832 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:3500 IOs, 1750703104 bytes 2023-08-26T13:38:35.919 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:3600 IOs, 1772991488 bytes 2023-08-26T13:38:36.080 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:3700 IOs, 1794478080 bytes 2023-08-26T13:38:36.158 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:3800 IOs, 1795104768 bytes 2023-08-26T13:38:36.388 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:3900 IOs, 1871045632 bytes 2023-08-26T13:38:36.567 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:4000 IOs, 1922865152 bytes 2023-08-26T13:38:36.648 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ error_code=124 2023-08-26T13:38:36.648 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ set -e 2023-08-26T13:38:36.649 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ '[' 124 -eq 124 ']' 2023-08-26T13:38:36.649 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ return 0 2023-08-26T13:38:36.649 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ wait_for_status_in_pool_dir cluster1 mirror test up+replaying 2023-08-26T13:38:36.649 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster1 2023-08-26T13:38:36.649 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T13:38:36.650 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=test 2023-08-26T13:38:36.650 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local state_pattern=up+replaying 2023-08-26T13:38:36.650 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local description_pattern= 2023-08-26T13:38:36.650 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local service_pattern= 2023-08-26T13:38:36.650 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-08-26T13:38:36.651 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ sleep 1 2023-08-26T13:38:37.650 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ test_status_in_pool_dir cluster1 mirror test up+replaying '' '' 2023-08-26T13:38:37.650 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster1 2023-08-26T13:38:37.651 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T13:38:37.651 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=test 2023-08-26T13:38:37.651 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local state_pattern=up+replaying 2023-08-26T13:38:37.652 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local description_pattern= 2023-08-26T13:38:37.652 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local service_pattern= 2023-08-26T13:38:37.652 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ mkfname cluster1-mirror-test.mirror_status 2023-08-26T13:38:37.652 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ echo cluster1-mirror-test.mirror_status 2023-08-26T13:38:37.653 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ sed -e 's|[/ ]|_|g' 2023-08-26T13:38:37.654 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local status_log=/tmp/tmp.TfDn6BcQ3N/cluster1-mirror-test.mirror_status 2023-08-26T13:38:37.654 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ CEPH_ARGS= 2023-08-26T13:38:37.654 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rbd --cluster cluster1 mirror image status mirror/test 2023-08-26T13:38:37.654 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ tee /tmp/tmp.TfDn6BcQ3N/cluster1-mirror-test.mirror_status 2023-08-26T13:38:37.705 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:test: 2023-08-26T13:38:37.705 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr: global_id: e2f2554d-659e-4467-b7f8-dddd7ccb0ee9 2023-08-26T13:38:37.705 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr: state: up+replaying 2023-08-26T13:38:37.705 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr: description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":533805568.0,"last_snapshot_bytes":535710720,"last_snapshot_sync_seconds":1,"local_snapshot_timestamp":1693057052,"remote_snapshot_timestamp":1693057052,"replay_state":"idle"} 2023-08-26T13:38:37.706 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr: service: mirror.2 on smithi124 2023-08-26T13:38:37.706 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr: last_update: 2023-08-26 13:38:15 2023-08-26T13:38:37.706 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr: peer_sites: 2023-08-26T13:38:37.706 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr: name: cluster2 2023-08-26T13:38:37.706 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr: state: up+stopped 2023-08-26T13:38:37.707 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr: description: local image is primary 2023-08-26T13:38:37.707 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr: last_update: 2023-08-26 13:38:15 2023-08-26T13:38:37.710 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ grep '^ state: .*up+replaying' /tmp/tmp.TfDn6BcQ3N/cluster1-mirror-test.mirror_status 2023-08-26T13:38:37.711 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: state: up+replaying 2023-08-26T13:38:37.711 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ grep '^ description: .*' /tmp/tmp.TfDn6BcQ3N/cluster1-mirror-test.mirror_status 2023-08-26T13:38:37.712 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":533805568.0,"last_snapshot_bytes":535710720,"last_snapshot_sync_seconds":1,"local_snapshot_timestamp":1693057052,"remote_snapshot_timestamp":1693057052,"replay_state":"idle"} 2023-08-26T13:38:37.712 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ '[' -n '' ']' 2023-08-26T13:38:37.712 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ echo up+replaying 2023-08-26T13:38:37.713 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ grep '^up+' 2023-08-26T13:38:37.713 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:up+replaying 2023-08-26T13:38:37.714 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ grep 'service: *mirror..* on ' /tmp/tmp.TfDn6BcQ3N/cluster1-mirror-test.mirror_status 2023-08-26T13:38:37.714 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: service: mirror.2 on smithi124 2023-08-26T13:38:37.714 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' /tmp/tmp.TfDn6BcQ3N/cluster1-mirror-test.mirror_status 2023-08-26T13:38:37.716 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local 'last_update=2023-08-26 13:38:15' 2023-08-26T13:38:37.716 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ test_mirror_pool_status_verbose cluster1 mirror test up+replaying '2023-08-26 13:38:15' 2023-08-26T13:38:37.716 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster1 2023-08-26T13:38:37.716 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T13:38:37.717 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=test 2023-08-26T13:38:37.717 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local state_pattern=up+replaying 2023-08-26T13:38:37.717 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local 'prev_last_update=2023-08-26 13:38:15' 2023-08-26T13:38:37.717 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ mkfname cluster1-mirror.mirror_status 2023-08-26T13:38:37.717 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ echo cluster1-mirror.mirror_status 2023-08-26T13:38:37.718 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ sed -e 's|[/ ]|_|g' 2023-08-26T13:38:37.718 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local status_log=/tmp/tmp.TfDn6BcQ3N/cluster1-mirror.mirror_status 2023-08-26T13:38:37.718 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2023-08-26T13:38:37.755 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:rbd: failed to query services: (13) Permission denied 2023-08-26T13:38:37.781 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local last_update state 2023-08-26T13:38:37.782 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update' 2023-08-26T13:38:37.783 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ last_update='2023-08-26 13:38:15' 2023-08-26T13:38:37.784 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state' 2023-08-26T13:38:37.785 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ state=up+replaying 2023-08-26T13:38:37.786 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ echo up+replaying 2023-08-26T13:38:37.786 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ grep up+replaying 2023-08-26T13:38:37.786 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:up+replaying 2023-08-26T13:38:37.787 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ return 0 2023-08-26T13:38:37.788 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ return 0 2023-08-26T13:38:37.788 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ snap_name=snap7 2023-08-26T13:38:37.788 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ create_snap cluster2 mirror test snap7 2023-08-26T13:38:37.788 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster2 2023-08-26T13:38:37.788 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T13:38:37.789 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=test 2023-08-26T13:38:37.789 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local snap_name=snap7 2023-08-26T13:38:37.789 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rbd --cluster cluster2 -p mirror snap create test@snap7 --debug-rbd=20 --debug-journaler=20 2023-08-26T13:38:58.472 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.0 with skip (0.3258579172339613) > weight (0.25) 2023-08-26T13:38:58.472 INFO:tasks.rbd_mirror_thrash:kill cluster1.client.mirror.1 2023-08-26T13:38:58.472 INFO:tasks.rbd_mirror.cluster1.client.mirror.1:Sent signal 15 2023-08-26T13:38:58.473 INFO:tasks.rbd_mirror_thrash:waiting for 5 secs before reviving daemons 2023-08-26T13:38:58.473 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi124.stderr:2023-08-26T13:38:58.472+0000 7f6c93879700 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.1 (PID: 107740) UID: 1000 2023-08-26T13:39:03.474 INFO:tasks.rbd_mirror_thrash:waiting for cluster1.client.mirror.1 2023-08-26T13:39:03.474 INFO:teuthology.orchestra.run:waiting for 600 2023-08-26T13:39:03.475 INFO:tasks.rbd_mirror_thrash:reviving cluster1.client.mirror.1 2023-08-26T13:39:03.475 INFO:tasks.rbd_mirror.cluster1.client.mirror.1:Restarting daemon 2023-08-26T13:39:03.475 DEBUG:teuthology.orchestra.run.smithi124:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.1 2023-08-26T13:39:03.478 INFO:tasks.rbd_mirror.cluster1.client.mirror.1:Started 2023-08-26T13:39:03.478 INFO:tasks.rbd_mirror_thrash:waiting for 113 secs before thrashing 2023-08-26T13:39:06.646 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ wait_for_image_replay_started cluster1 mirror test 2023-08-26T13:39:06.647 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster1 2023-08-26T13:39:06.647 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T13:39:06.647 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=test 2023-08-26T13:39:06.647 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ wait_for_image_replay_state cluster1 mirror test started 2023-08-26T13:39:06.648 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster1 2023-08-26T13:39:06.648 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T13:39:06.648 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=test 2023-08-26T13:39:06.648 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local state=started 2023-08-26T13:39:06.648 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local s 2023-08-26T13:39:06.649 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-08-26T13:39:06.649 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ sleep 1 2023-08-26T13:39:07.648 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ test_image_replay_state cluster1 mirror test started 2023-08-26T13:39:07.649 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster1 2023-08-26T13:39:07.649 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T13:39:07.649 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=test 2023-08-26T13:39:07.649 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local test_state=started 2023-08-26T13:39:07.649 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local status_result 2023-08-26T13:39:07.650 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local current_state=stopped 2023-08-26T13:39:07.650 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ admin_daemons cluster1 rbd mirror status mirror/test 2023-08-26T13:39:07.650 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local cluster_instance=cluster1 2023-08-26T13:39:07.650 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ grep -i state 2023-08-26T13:39:07.650 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ shift 2023-08-26T13:39:07.651 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local cluster=cluster1 2023-08-26T13:39:07.651 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local instance=cluster1 2023-08-26T13:39:07.651 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local loop_instance 2023-08-26T13:39:07.651 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-08-26T13:39:07.651 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ sleep 0 2023-08-26T13:39:07.654 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ '[' cluster1 '!=' cluster1 ']' 2023-08-26T13:39:07.654 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ seq 0 3 2023-08-26T13:39:07.654 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-08-26T13:39:07.654 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test 2023-08-26T13:39:07.654 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local cluster=cluster1:0 2023-08-26T13:39:07.655 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ shift 2023-08-26T13:39:07.655 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local instance 2023-08-26T13:39:07.655 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ set_cluster_instance cluster1:0 cluster instance 2023-08-26T13:39:07.655 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local val=cluster1:0 2023-08-26T13:39:07.655 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local cluster_var_name=cluster 2023-08-26T13:39:07.656 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local instance_var_name=instance 2023-08-26T13:39:07.656 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ cluster=cluster1 2023-08-26T13:39:07.656 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ instance=0 2023-08-26T13:39:07.656 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ '[' 0 = cluster1:0 ']' 2023-08-26T13:39:07.656 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ eval cluster=cluster1 2023-08-26T13:39:07.656 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ cluster=cluster1 2023-08-26T13:39:07.657 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ eval instance=0 2023-08-26T13:39:07.657 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ instance=0 2023-08-26T13:39:07.657 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ daemon_asok_file cluster1:0 cluster1 2023-08-26T13:39:07.657 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ local local_cluster=cluster1:0 2023-08-26T13:39:07.657 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ local cluster=cluster1 2023-08-26T13:39:07.657 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ local instance 2023-08-26T13:39:07.658 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2023-08-26T13:39:07.659 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ local val=cluster1:0 2023-08-26T13:39:07.659 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ local cluster_var_name=local_cluster 2023-08-26T13:39:07.660 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ local instance_var_name=instance 2023-08-26T13:39:07.660 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ cluster=cluster1 2023-08-26T13:39:07.660 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ instance=0 2023-08-26T13:39:07.660 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ '[' 0 = cluster1:0 ']' 2023-08-26T13:39:07.660 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ eval local_cluster=cluster1 2023-08-26T13:39:07.661 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++++ local_cluster=cluster1 2023-08-26T13:39:07.661 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ eval instance=0 2023-08-26T13:39:07.661 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++++ instance=0 2023-08-26T13:39:07.661 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2023-08-26T13:39:07.675 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-08-26T13:39:07.675 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-08-26T13:39:07.676 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-08-26T13:39:07.676 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2023-08-26T13:39:07.774 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:Invalid command: unused arguments: ['mirror/test'] 2023-08-26T13:39:07.774 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:rbd mirror status : get status for rbd mirror 2023-08-26T13:39:07.775 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:admin_socket: invalid command 2023-08-26T13:39:07.778 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-08-26T13:39:07.779 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/test 2023-08-26T13:39:07.779 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local cluster=cluster1:1 2023-08-26T13:39:07.779 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ shift 2023-08-26T13:39:07.780 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local instance 2023-08-26T13:39:07.780 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ set_cluster_instance cluster1:1 cluster instance 2023-08-26T13:39:07.780 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local val=cluster1:1 2023-08-26T13:39:07.780 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local cluster_var_name=cluster 2023-08-26T13:39:07.781 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local instance_var_name=instance 2023-08-26T13:39:07.781 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ cluster=cluster1 2023-08-26T13:39:07.781 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ instance=1 2023-08-26T13:39:07.781 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ '[' 1 = cluster1:1 ']' 2023-08-26T13:39:07.782 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ eval cluster=cluster1 2023-08-26T13:39:07.782 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ cluster=cluster1 2023-08-26T13:39:07.782 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ eval instance=1 2023-08-26T13:39:07.782 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ instance=1 2023-08-26T13:39:07.783 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ daemon_asok_file cluster1:1 cluster1 2023-08-26T13:39:07.783 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ local local_cluster=cluster1:1 2023-08-26T13:39:07.783 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ local cluster=cluster1 2023-08-26T13:39:07.783 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ local instance 2023-08-26T13:39:07.783 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2023-08-26T13:39:07.784 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ local val=cluster1:1 2023-08-26T13:39:07.784 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ local cluster_var_name=local_cluster 2023-08-26T13:39:07.784 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ local instance_var_name=instance 2023-08-26T13:39:07.784 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ cluster=cluster1 2023-08-26T13:39:07.784 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ instance=1 2023-08-26T13:39:07.784 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ '[' 1 = cluster1:1 ']' 2023-08-26T13:39:07.785 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ eval local_cluster=cluster1 2023-08-26T13:39:07.785 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++++ local_cluster=cluster1 2023-08-26T13:39:07.785 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ eval instance=1 2023-08-26T13:39:07.785 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++++ instance=1 2023-08-26T13:39:07.785 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2023-08-26T13:39:07.796 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-08-26T13:39:07.796 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-08-26T13:39:07.796 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-08-26T13:39:07.796 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/test 2023-08-26T13:39:07.890 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:Invalid command: unused arguments: ['mirror/test'] 2023-08-26T13:39:07.890 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:rbd mirror status : get status for rbd mirror 2023-08-26T13:39:07.890 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:admin_socket: invalid command 2023-08-26T13:39:07.894 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-08-26T13:39:07.894 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/test 2023-08-26T13:39:07.894 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local cluster=cluster1:2 2023-08-26T13:39:07.894 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ shift 2023-08-26T13:39:07.894 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local instance 2023-08-26T13:39:07.895 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ set_cluster_instance cluster1:2 cluster instance 2023-08-26T13:39:07.895 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local val=cluster1:2 2023-08-26T13:39:07.895 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local cluster_var_name=cluster 2023-08-26T13:39:07.895 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local instance_var_name=instance 2023-08-26T13:39:07.895 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ cluster=cluster1 2023-08-26T13:39:07.896 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ instance=2 2023-08-26T13:39:07.896 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ '[' 2 = cluster1:2 ']' 2023-08-26T13:39:07.896 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ eval cluster=cluster1 2023-08-26T13:39:07.896 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ cluster=cluster1 2023-08-26T13:39:07.896 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ eval instance=2 2023-08-26T13:39:07.897 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ instance=2 2023-08-26T13:39:07.897 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ daemon_asok_file cluster1:2 cluster1 2023-08-26T13:39:07.897 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ local local_cluster=cluster1:2 2023-08-26T13:39:07.897 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ local cluster=cluster1 2023-08-26T13:39:07.897 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ local instance 2023-08-26T13:39:07.897 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2023-08-26T13:39:07.898 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ local val=cluster1:2 2023-08-26T13:39:07.898 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ local cluster_var_name=local_cluster 2023-08-26T13:39:07.898 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ local instance_var_name=instance 2023-08-26T13:39:07.898 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ cluster=cluster1 2023-08-26T13:39:07.898 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ instance=2 2023-08-26T13:39:07.899 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ '[' 2 = cluster1:2 ']' 2023-08-26T13:39:07.899 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ eval local_cluster=cluster1 2023-08-26T13:39:07.899 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++++ local_cluster=cluster1 2023-08-26T13:39:07.899 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ eval instance=2 2023-08-26T13:39:07.899 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++++ instance=2 2023-08-26T13:39:07.900 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2023-08-26T13:39:07.912 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-08-26T13:39:07.913 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-08-26T13:39:07.913 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-08-26T13:39:07.913 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/test 2023-08-26T13:39:08.016 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ return 0 2023-08-26T13:39:08.017 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ status_result=' "state": "Replaying"' 2023-08-26T13:39:08.017 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ echo ' "state": "Replaying"' 2023-08-26T13:39:08.018 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ grep -i Replaying 2023-08-26T13:39:08.018 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: "state": "Replaying" 2023-08-26T13:39:08.019 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ current_state=started 2023-08-26T13:39:08.020 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ test started = started 2023-08-26T13:39:08.020 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ return 0 2023-08-26T13:39:08.020 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror test 2023-08-26T13:39:08.020 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local local_cluster=cluster1 2023-08-26T13:39:08.020 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster2 2023-08-26T13:39:08.021 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T13:39:08.021 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=test 2023-08-26T13:39:08.021 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ '[' snapshot = journal ']' 2023-08-26T13:39:08.021 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ '[' snapshot = snapshot ']' 2023-08-26T13:39:08.021 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror test 2023-08-26T13:39:08.021 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local local_cluster=cluster1 2023-08-26T13:39:08.022 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster2 2023-08-26T13:39:08.022 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T13:39:08.022 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=test 2023-08-26T13:39:08.022 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ mkfname cluster2-mirror-test.status 2023-08-26T13:39:08.022 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ echo cluster2-mirror-test.status 2023-08-26T13:39:08.022 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ sed -e 's|[/ ]|_|g' 2023-08-26T13:39:08.024 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local status_log=/tmp/tmp.TfDn6BcQ3N/cluster2-mirror-test.status 2023-08-26T13:39:08.024 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ mkfname cluster1-mirror-test.status 2023-08-26T13:39:08.024 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ echo cluster1-mirror-test.status 2023-08-26T13:39:08.024 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ sed -e 's|[/ ]|_|g' 2023-08-26T13:39:08.024 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local local_status_log=/tmp/tmp.TfDn6BcQ3N/cluster1-mirror-test.status 2023-08-26T13:39:08.025 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ mirror_image_snapshot cluster2 mirror test 2023-08-26T13:39:08.025 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster2 2023-08-26T13:39:08.025 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T13:39:08.025 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=test 2023-08-26T13:39:08.025 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/test 2023-08-26T13:39:09.643 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:Snapshot ID: 29 2023-08-26T13:39:09.661 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ get_newest_mirror_snapshot cluster2 mirror test /tmp/tmp.TfDn6BcQ3N/cluster2-mirror-test.status 2023-08-26T13:39:09.661 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster2 2023-08-26T13:39:09.661 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T13:39:09.661 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=test 2023-08-26T13:39:09.661 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local log=/tmp/tmp.TfDn6BcQ3N/cluster2-mirror-test.status 2023-08-26T13:39:09.662 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rbd --cluster cluster2 snap list --all mirror/test --format xml 2023-08-26T13:39:09.662 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-08-26T13:39:09.731 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ xmlstarlet sel -t -v //snapshot/id 2023-08-26T13:39:09.735 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local snapshot_id=29 2023-08-26T13:39:09.735 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ true 2023-08-26T13:39:09.735 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-08-26T13:39:09.735 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ sleep 0.2 2023-08-26T13:39:09.936 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.TfDn6BcQ3N/cluster1-mirror-test.status 2023-08-26T13:39:09.937 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster1 2023-08-26T13:39:09.937 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T13:39:09.937 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=test 2023-08-26T13:39:09.937 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local log=/tmp/tmp.TfDn6BcQ3N/cluster1-mirror-test.status 2023-08-26T13:39:09.937 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2023-08-26T13:39:09.937 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-08-26T13:39:09.983 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ true 2023-08-26T13:39:09.983 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-08-26T13:39:09.984 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:-:1.1: Document is empty 2023-08-26T13:39:09.984 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr: 2023-08-26T13:39:09.984 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:^ 2023-08-26T13:39:09.985 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local primary_snapshot_id= 2023-08-26T13:39:09.985 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ test 29 = '' 2023-08-26T13:39:09.986 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-08-26T13:39:09.986 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ sleep 0.4 2023-08-26T13:39:10.386 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.TfDn6BcQ3N/cluster1-mirror-test.status 2023-08-26T13:39:10.387 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster1 2023-08-26T13:39:10.387 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T13:39:10.387 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=test 2023-08-26T13:39:10.388 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local log=/tmp/tmp.TfDn6BcQ3N/cluster1-mirror-test.status 2023-08-26T13:39:10.388 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2023-08-26T13:39:10.388 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-08-26T13:39:10.435 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ true 2023-08-26T13:39:10.435 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-08-26T13:39:10.436 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:-:1.1: Document is empty 2023-08-26T13:39:10.436 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr: 2023-08-26T13:39:10.437 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:^ 2023-08-26T13:39:10.437 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local primary_snapshot_id= 2023-08-26T13:39:10.438 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ test 29 = '' 2023-08-26T13:39:10.438 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-08-26T13:39:10.438 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ sleep 0.8 2023-08-26T13:39:11.238 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.TfDn6BcQ3N/cluster1-mirror-test.status 2023-08-26T13:39:11.238 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster1 2023-08-26T13:39:11.239 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T13:39:11.239 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=test 2023-08-26T13:39:11.239 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local log=/tmp/tmp.TfDn6BcQ3N/cluster1-mirror-test.status 2023-08-26T13:39:11.239 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2023-08-26T13:39:11.239 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-08-26T13:39:11.408 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ true 2023-08-26T13:39:11.409 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-08-26T13:39:11.410 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:-:1.1: Document is empty 2023-08-26T13:39:11.410 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr: 2023-08-26T13:39:11.410 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:^ 2023-08-26T13:39:11.411 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local primary_snapshot_id= 2023-08-26T13:39:11.411 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ test 29 = '' 2023-08-26T13:39:11.411 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-08-26T13:39:11.411 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ sleep 1.6 2023-08-26T13:39:13.011 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.TfDn6BcQ3N/cluster1-mirror-test.status 2023-08-26T13:39:13.012 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster1 2023-08-26T13:39:13.012 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T13:39:13.012 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=test 2023-08-26T13:39:13.012 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local log=/tmp/tmp.TfDn6BcQ3N/cluster1-mirror-test.status 2023-08-26T13:39:13.013 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2023-08-26T13:39:13.013 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-08-26T13:39:13.062 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-08-26T13:39:13.064 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local primary_snapshot_id=29 2023-08-26T13:39:13.064 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ test 29 = 29 2023-08-26T13:39:13.064 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ return 0 2023-08-26T13:39:13.065 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ wait_for_snap_present cluster1 mirror test snap7 2023-08-26T13:39:13.065 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster1 2023-08-26T13:39:13.065 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T13:39:13.065 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=test 2023-08-26T13:39:13.065 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local snap_name=snap7 2023-08-26T13:39:13.066 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local s 2023-08-26T13:39:13.066 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2023-08-26T13:39:13.066 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ sleep 1 2023-08-26T13:39:14.065 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rbd --cluster cluster1 info mirror/test@snap7 2023-08-26T13:39:14.314 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:rbd image 'test': 2023-08-26T13:39:14.314 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: size 512 MiB in 128 objects 2023-08-26T13:39:14.315 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: order 22 (4 MiB objects) 2023-08-26T13:39:14.315 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: snapshot_count: 14 2023-08-26T13:39:14.315 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: id: 10ab632cdafc 2023-08-26T13:39:14.315 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: block_name_prefix: rbd_data.10ab632cdafc 2023-08-26T13:39:14.315 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: format: 2 2023-08-26T13:39:14.315 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: features: layering, exclusive-lock, non-primary 2023-08-26T13:39:14.316 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: op_features: 2023-08-26T13:39:14.316 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: flags: 2023-08-26T13:39:14.316 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: create_timestamp: Sat Aug 26 13:33:17 2023 2023-08-26T13:39:14.316 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: access_timestamp: Sat Aug 26 13:37:38 2023 2023-08-26T13:39:14.316 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: modify_timestamp: Sat Aug 26 13:33:17 2023 2023-08-26T13:39:14.317 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: protected: False 2023-08-26T13:39:14.317 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: mirroring state: enabled 2023-08-26T13:39:14.317 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: mirroring mode: snapshot 2023-08-26T13:39:14.317 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: mirroring global id: e2f2554d-659e-4467-b7f8-dddd7ccb0ee9 2023-08-26T13:39:14.317 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: mirroring primary: false 2023-08-26T13:39:14.319 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ return 0 2023-08-26T13:39:14.319 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ '[' -n snap6-clean ']' 2023-08-26T13:39:14.319 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ compare_image_snaps mirror test snap6-clean 2023-08-26T13:39:14.319 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T13:39:14.319 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=test 2023-08-26T13:39:14.320 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local snap_name=snap6-clean 2023-08-26T13:39:14.320 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local ret=0 2023-08-26T13:39:14.320 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local rmt_export=/tmp/tmp.TfDn6BcQ3N/cluster2-mirror-test.export 2023-08-26T13:39:14.320 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local loc_export=/tmp/tmp.TfDn6BcQ3N/cluster1-mirror-test.export 2023-08-26T13:39:14.320 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rm -f /tmp/tmp.TfDn6BcQ3N/cluster2-mirror-test.export /tmp/tmp.TfDn6BcQ3N/cluster1-mirror-test.export 2023-08-26T13:39:14.320 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rbd --cluster cluster2 -p mirror export test@snap6-clean /tmp/tmp.TfDn6BcQ3N/cluster2-mirror-test.export 2023-08-26T13:39:14.901 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2023-08-26T13:39:14.911 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rbd --cluster cluster1 -p mirror export test@snap6-clean /tmp/tmp.TfDn6BcQ3N/cluster1-mirror-test.export 2023-08-26T13:39:15.708 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2023-08-26T13:39:15.717 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ cmp /tmp/tmp.TfDn6BcQ3N/cluster2-mirror-test.export /tmp/tmp.TfDn6BcQ3N/cluster1-mirror-test.export 2023-08-26T13:39:15.932 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rm -f /tmp/tmp.TfDn6BcQ3N/cluster2-mirror-test.export /tmp/tmp.TfDn6BcQ3N/cluster1-mirror-test.export 2023-08-26T13:39:16.074 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ return 0 2023-08-26T13:39:16.074 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ compare_image_snaps mirror test snap7 2023-08-26T13:39:16.074 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T13:39:16.074 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=test 2023-08-26T13:39:16.075 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local snap_name=snap7 2023-08-26T13:39:16.075 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local ret=0 2023-08-26T13:39:16.075 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local rmt_export=/tmp/tmp.TfDn6BcQ3N/cluster2-mirror-test.export 2023-08-26T13:39:16.075 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local loc_export=/tmp/tmp.TfDn6BcQ3N/cluster1-mirror-test.export 2023-08-26T13:39:16.075 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rm -f /tmp/tmp.TfDn6BcQ3N/cluster2-mirror-test.export /tmp/tmp.TfDn6BcQ3N/cluster1-mirror-test.export 2023-08-26T13:39:16.076 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rbd --cluster cluster2 -p mirror export test@snap7 /tmp/tmp.TfDn6BcQ3N/cluster2-mirror-test.export 2023-08-26T13:39:16.657 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2023-08-26T13:39:16.667 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rbd --cluster cluster1 -p mirror export test@snap7 /tmp/tmp.TfDn6BcQ3N/cluster1-mirror-test.export 2023-08-26T13:39:17.396 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2023-08-26T13:39:17.405 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ cmp /tmp/tmp.TfDn6BcQ3N/cluster2-mirror-test.export /tmp/tmp.TfDn6BcQ3N/cluster1-mirror-test.export 2023-08-26T13:39:17.621 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rm -f /tmp/tmp.TfDn6BcQ3N/cluster2-mirror-test.export /tmp/tmp.TfDn6BcQ3N/cluster1-mirror-test.export 2023-08-26T13:39:17.764 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ return 0 2023-08-26T13:39:17.764 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ clean_snap_name=snap7-clean 2023-08-26T13:39:17.764 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ create_snap cluster2 mirror test snap7-clean 2023-08-26T13:39:17.764 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster2 2023-08-26T13:39:17.765 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T13:39:17.765 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=test 2023-08-26T13:39:17.765 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local snap_name=snap7-clean 2023-08-26T13:39:17.765 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rbd --cluster cluster2 -p mirror snap create test@snap7-clean --debug-rbd=20 --debug-journaler=20 2023-08-26T13:39:18.644 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ for i in `seq 1 10` 2023-08-26T13:39:18.644 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ stress_write_image cluster2 mirror test 2023-08-26T13:39:18.645 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster2 2023-08-26T13:39:18.645 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T13:39:18.645 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=test 2023-08-26T13:39:18.645 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2023-08-26T13:39:18.646 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local duration=12 2023-08-26T13:39:18.646 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ set +e 2023-08-26T13:39:18.646 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ timeout 12s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror test --debug-rbd=20 --debug-journaler=20 2023-08-26T13:39:18.861 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:100 IOs, 24071168 bytes 2023-08-26T13:39:18.985 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:200 IOs, 49441792 bytes 2023-08-26T13:39:19.273 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:300 IOs, 148497408 bytes 2023-08-26T13:39:19.478 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:400 IOs, 209897472 bytes 2023-08-26T13:39:19.548 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:500 IOs, 210507776 bytes 2023-08-26T13:39:19.746 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:600 IOs, 293539840 bytes 2023-08-26T13:39:19.960 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:700 IOs, 346677248 bytes 2023-08-26T13:39:20.054 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:800 IOs, 360563712 bytes 2023-08-26T13:39:20.244 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:900 IOs, 422185984 bytes 2023-08-26T13:39:20.468 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:1000 IOs, 503921664 bytes 2023-08-26T13:39:20.622 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:1100 IOs, 544530432 bytes 2023-08-26T13:39:20.840 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:1200 IOs, 609916928 bytes 2023-08-26T13:39:20.986 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:1300 IOs, 651491328 bytes 2023-08-26T13:39:21.140 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:1400 IOs, 695853056 bytes 2023-08-26T13:39:21.306 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:1500 IOs, 755718144 bytes 2023-08-26T13:39:21.459 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:1600 IOs, 796677120 bytes 2023-08-26T13:39:21.677 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:1700 IOs, 862991360 bytes 2023-08-26T13:39:21.833 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:1800 IOs, 901051392 bytes 2023-08-26T13:39:21.963 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:1900 IOs, 930853888 bytes 2023-08-26T13:39:22.103 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:2000 IOs, 963147776 bytes 2023-08-26T13:39:22.316 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:2100 IOs, 1033657344 bytes 2023-08-26T13:39:22.427 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:2200 IOs, 1051168768 bytes 2023-08-26T13:39:22.514 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:2300 IOs, 1061995520 bytes 2023-08-26T13:39:22.607 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:2400 IOs, 1067032576 bytes 2023-08-26T13:39:22.778 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:2500 IOs, 1120277504 bytes 2023-08-26T13:39:22.855 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:2600 IOs, 1120909312 bytes 2023-08-26T13:39:22.930 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:2700 IOs, 1127451648 bytes 2023-08-26T13:39:23.045 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:2800 IOs, 1154132992 bytes 2023-08-26T13:39:23.187 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:2900 IOs, 1193827328 bytes 2023-08-26T13:39:23.327 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:3000 IOs, 1242221568 bytes 2023-08-26T13:39:23.481 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:3100 IOs, 1264952320 bytes 2023-08-26T13:39:23.591 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:3200 IOs, 1300364288 bytes 2023-08-26T13:39:23.862 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:3300 IOs, 1331164160 bytes 2023-08-26T13:39:23.987 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:3400 IOs, 1354113024 bytes 2023-08-26T13:39:24.134 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:3500 IOs, 1394335744 bytes 2023-08-26T13:39:24.278 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:3600 IOs, 1422529536 bytes 2023-08-26T13:39:24.417 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:3700 IOs, 1455082496 bytes 2023-08-26T13:39:24.596 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:3800 IOs, 1499425792 bytes 2023-08-26T13:39:24.708 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:3900 IOs, 1511710720 bytes 2023-08-26T13:39:24.847 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:4000 IOs, 1554999296 bytes 2023-08-26T13:39:25.032 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:4100 IOs, 1602016256 bytes 2023-08-26T13:39:25.186 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:4200 IOs, 1641711616 bytes 2023-08-26T13:39:25.338 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:4300 IOs, 1697025024 bytes 2023-08-26T13:39:25.489 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:4400 IOs, 1730885632 bytes 2023-08-26T13:39:25.692 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:4500 IOs, 1788822528 bytes 2023-08-26T13:39:25.830 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:4600 IOs, 1822792704 bytes 2023-08-26T13:39:26.000 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:4700 IOs, 1882772480 bytes 2023-08-26T13:39:26.220 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:4800 IOs, 1942181888 bytes 2023-08-26T13:39:26.385 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:4900 IOs, 1987563520 bytes 2023-08-26T13:39:26.515 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:5000 IOs, 2015842304 bytes 2023-08-26T13:39:26.658 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:5100 IOs, 2049116160 bytes 2023-08-26T13:39:26.906 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:5200 IOs, 2146633728 bytes 2023-08-26T13:39:27.132 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:5300 IOs, 2206645248 bytes 2023-08-26T13:39:27.261 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:5400 IOs, 2232027136 bytes 2023-08-26T13:39:27.351 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:5500 IOs, 2261091328 bytes 2023-08-26T13:39:27.568 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:5600 IOs, 2318066688 bytes 2023-08-26T13:39:27.715 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:5700 IOs, 2348606464 bytes 2023-08-26T13:39:27.897 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:5800 IOs, 2413259776 bytes 2023-08-26T13:39:28.129 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:5900 IOs, 2485000192 bytes 2023-08-26T13:39:28.292 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:6000 IOs, 2538204160 bytes 2023-08-26T13:39:28.505 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:6100 IOs, 2589590528 bytes 2023-08-26T13:39:28.696 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:6200 IOs, 2650018816 bytes 2023-08-26T13:39:28.826 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:6300 IOs, 2672569344 bytes 2023-08-26T13:39:28.969 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:6400 IOs, 2707089408 bytes 2023-08-26T13:39:29.196 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:6500 IOs, 2786013184 bytes 2023-08-26T13:39:29.337 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:6600 IOs, 2813327360 bytes 2023-08-26T13:39:29.444 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:6700 IOs, 2829933568 bytes 2023-08-26T13:39:29.644 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:6800 IOs, 2888309760 bytes 2023-08-26T13:39:29.819 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:6900 IOs, 2931730432 bytes 2023-08-26T13:39:29.915 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:7000 IOs, 2934980608 bytes 2023-08-26T13:39:30.007 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:7100 IOs, 2949661696 bytes 2023-08-26T13:39:30.128 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:7200 IOs, 2983982080 bytes 2023-08-26T13:39:30.395 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:7300 IOs, 3068592128 bytes 2023-08-26T13:39:30.602 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:7400 IOs, 3131869184 bytes 2023-08-26T13:39:30.654 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ error_code=124 2023-08-26T13:39:30.654 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ set -e 2023-08-26T13:39:30.654 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ '[' 124 -eq 124 ']' 2023-08-26T13:39:30.654 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ return 0 2023-08-26T13:39:30.655 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ wait_for_status_in_pool_dir cluster1 mirror test up+replaying 2023-08-26T13:39:30.655 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster1 2023-08-26T13:39:30.655 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T13:39:30.655 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=test 2023-08-26T13:39:30.655 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local state_pattern=up+replaying 2023-08-26T13:39:30.655 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local description_pattern= 2023-08-26T13:39:30.656 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local service_pattern= 2023-08-26T13:39:30.656 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-08-26T13:39:30.656 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ sleep 1 2023-08-26T13:39:31.655 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ test_status_in_pool_dir cluster1 mirror test up+replaying '' '' 2023-08-26T13:39:31.655 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster1 2023-08-26T13:39:31.656 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T13:39:31.656 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=test 2023-08-26T13:39:31.656 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local state_pattern=up+replaying 2023-08-26T13:39:31.656 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local description_pattern= 2023-08-26T13:39:31.656 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local service_pattern= 2023-08-26T13:39:31.657 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ mkfname cluster1-mirror-test.mirror_status 2023-08-26T13:39:31.657 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ echo cluster1-mirror-test.mirror_status 2023-08-26T13:39:31.657 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ sed -e 's|[/ ]|_|g' 2023-08-26T13:39:31.658 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local status_log=/tmp/tmp.TfDn6BcQ3N/cluster1-mirror-test.mirror_status 2023-08-26T13:39:31.658 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ CEPH_ARGS= 2023-08-26T13:39:31.658 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rbd --cluster cluster1 mirror image status mirror/test 2023-08-26T13:39:31.658 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ tee /tmp/tmp.TfDn6BcQ3N/cluster1-mirror-test.mirror_status 2023-08-26T13:39:31.712 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:test: 2023-08-26T13:39:31.713 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr: global_id: e2f2554d-659e-4467-b7f8-dddd7ccb0ee9 2023-08-26T13:39:31.713 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr: state: up+replaying 2023-08-26T13:39:31.713 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr: description: replaying, {"bytes_per_second":17732369.07,"bytes_per_snapshot":521703424.0,"last_snapshot_bytes":531971072,"last_snapshot_sync_seconds":1,"local_snapshot_timestamp":1693057148,"remote_snapshot_timestamp":1693057148,"replay_state":"idle"} 2023-08-26T13:39:31.714 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr: service: mirror.2 on smithi124 2023-08-26T13:39:31.714 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr: last_update: 2023-08-26 13:39:15 2023-08-26T13:39:31.714 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr: peer_sites: 2023-08-26T13:39:31.715 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr: name: cluster2 2023-08-26T13:39:31.715 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr: state: up+stopped 2023-08-26T13:39:31.715 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr: description: local image is primary 2023-08-26T13:39:31.716 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr: last_update: 2023-08-26 13:39:15 2023-08-26T13:39:31.717 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ grep '^ state: .*up+replaying' /tmp/tmp.TfDn6BcQ3N/cluster1-mirror-test.mirror_status 2023-08-26T13:39:31.718 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: state: up+replaying 2023-08-26T13:39:31.719 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ grep '^ description: .*' /tmp/tmp.TfDn6BcQ3N/cluster1-mirror-test.mirror_status 2023-08-26T13:39:31.719 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: description: replaying, {"bytes_per_second":17732369.07,"bytes_per_snapshot":521703424.0,"last_snapshot_bytes":531971072,"last_snapshot_sync_seconds":1,"local_snapshot_timestamp":1693057148,"remote_snapshot_timestamp":1693057148,"replay_state":"idle"} 2023-08-26T13:39:31.720 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ '[' -n '' ']' 2023-08-26T13:39:31.721 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ echo up+replaying 2023-08-26T13:39:31.721 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ grep '^up+' 2023-08-26T13:39:31.721 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:up+replaying 2023-08-26T13:39:31.722 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ grep 'service: *mirror..* on ' /tmp/tmp.TfDn6BcQ3N/cluster1-mirror-test.mirror_status 2023-08-26T13:39:31.722 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: service: mirror.2 on smithi124 2023-08-26T13:39:31.724 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' /tmp/tmp.TfDn6BcQ3N/cluster1-mirror-test.mirror_status 2023-08-26T13:39:31.724 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local 'last_update=2023-08-26 13:39:15' 2023-08-26T13:39:31.724 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ test_mirror_pool_status_verbose cluster1 mirror test up+replaying '2023-08-26 13:39:15' 2023-08-26T13:39:31.724 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster1 2023-08-26T13:39:31.725 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T13:39:31.725 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=test 2023-08-26T13:39:31.725 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local state_pattern=up+replaying 2023-08-26T13:39:31.725 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local 'prev_last_update=2023-08-26 13:39:15' 2023-08-26T13:39:31.725 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ mkfname cluster1-mirror.mirror_status 2023-08-26T13:39:31.726 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ echo cluster1-mirror.mirror_status 2023-08-26T13:39:31.726 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ sed -e 's|[/ ]|_|g' 2023-08-26T13:39:31.726 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local status_log=/tmp/tmp.TfDn6BcQ3N/cluster1-mirror.mirror_status 2023-08-26T13:39:31.726 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2023-08-26T13:39:31.762 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:rbd: failed to query services: (13) Permission denied 2023-08-26T13:39:31.786 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local last_update state 2023-08-26T13:39:31.786 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update' 2023-08-26T13:39:31.788 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ last_update='2023-08-26 13:39:15' 2023-08-26T13:39:31.788 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state' 2023-08-26T13:39:31.790 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ state=up+replaying 2023-08-26T13:39:31.790 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ echo up+replaying 2023-08-26T13:39:31.790 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ grep up+replaying 2023-08-26T13:39:31.791 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:up+replaying 2023-08-26T13:39:31.792 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ return 0 2023-08-26T13:39:31.792 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ return 0 2023-08-26T13:39:31.792 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ snap_name=snap8 2023-08-26T13:39:31.792 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ create_snap cluster2 mirror test snap8 2023-08-26T13:39:31.793 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster2 2023-08-26T13:39:31.793 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T13:39:31.793 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=test 2023-08-26T13:39:31.793 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local snap_name=snap8 2023-08-26T13:39:31.793 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rbd --cluster cluster2 -p mirror snap create test@snap8 --debug-rbd=20 --debug-journaler=20 2023-08-26T13:39:43.469 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.0 with skip (0.8162584115424739) > weight (0.25) 2023-08-26T13:39:43.469 INFO:tasks.rbd_mirror_thrash:kill cluster2.client.mirror.1 2023-08-26T13:39:43.470 INFO:tasks.rbd_mirror.cluster2.client.mirror.1:Sent signal 15 2023-08-26T13:39:43.470 INFO:tasks.rbd_mirror_thrash:waiting for 4 secs before reviving daemons 2023-08-26T13:39:43.470 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi124.stderr:2023-08-26T13:39:43.469+0000 7ffb3e12e700 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.1 (PID: 110671) UID: 1000 2023-08-26T13:39:47.471 INFO:tasks.rbd_mirror_thrash:waiting for cluster2.client.mirror.1 2023-08-26T13:39:47.472 INFO:teuthology.orchestra.run:waiting for 600 2023-08-26T13:39:47.472 INFO:tasks.rbd_mirror_thrash:reviving cluster2.client.mirror.1 2023-08-26T13:39:47.472 INFO:tasks.rbd_mirror.cluster2.client.mirror.1:Restarting daemon 2023-08-26T13:39:47.473 DEBUG:teuthology.orchestra.run.smithi124:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.1 2023-08-26T13:39:47.476 INFO:tasks.rbd_mirror.cluster2.client.mirror.1:Started 2023-08-26T13:39:47.476 INFO:tasks.rbd_mirror_thrash:waiting for 92 secs before thrashing 2023-08-26T13:40:00.677 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ wait_for_image_replay_started cluster1 mirror test 2023-08-26T13:40:00.678 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster1 2023-08-26T13:40:00.678 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T13:40:00.678 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=test 2023-08-26T13:40:00.678 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ wait_for_image_replay_state cluster1 mirror test started 2023-08-26T13:40:00.678 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster1 2023-08-26T13:40:00.679 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T13:40:00.679 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=test 2023-08-26T13:40:00.679 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local state=started 2023-08-26T13:40:00.679 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local s 2023-08-26T13:40:00.679 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-08-26T13:40:00.680 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ sleep 1 2023-08-26T13:40:01.678 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ test_image_replay_state cluster1 mirror test started 2023-08-26T13:40:01.679 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster1 2023-08-26T13:40:01.679 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T13:40:01.679 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=test 2023-08-26T13:40:01.679 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local test_state=started 2023-08-26T13:40:01.680 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local status_result 2023-08-26T13:40:01.680 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local current_state=stopped 2023-08-26T13:40:01.683 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ admin_daemons cluster1 rbd mirror status mirror/test 2023-08-26T13:40:01.683 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local cluster_instance=cluster1 2023-08-26T13:40:01.683 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ shift 2023-08-26T13:40:01.683 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ grep -i state 2023-08-26T13:40:01.683 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local cluster=cluster1 2023-08-26T13:40:01.684 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local instance=cluster1 2023-08-26T13:40:01.684 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local loop_instance 2023-08-26T13:40:01.684 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-08-26T13:40:01.684 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ sleep 0 2023-08-26T13:40:01.684 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ '[' cluster1 '!=' cluster1 ']' 2023-08-26T13:40:01.685 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ seq 0 3 2023-08-26T13:40:01.685 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-08-26T13:40:01.685 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test 2023-08-26T13:40:01.686 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local cluster=cluster1:0 2023-08-26T13:40:01.686 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ shift 2023-08-26T13:40:01.686 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local instance 2023-08-26T13:40:01.686 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ set_cluster_instance cluster1:0 cluster instance 2023-08-26T13:40:01.686 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local val=cluster1:0 2023-08-26T13:40:01.687 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local cluster_var_name=cluster 2023-08-26T13:40:01.687 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local instance_var_name=instance 2023-08-26T13:40:01.687 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ cluster=cluster1 2023-08-26T13:40:01.687 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ instance=0 2023-08-26T13:40:01.687 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ '[' 0 = cluster1:0 ']' 2023-08-26T13:40:01.688 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ eval cluster=cluster1 2023-08-26T13:40:01.688 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ cluster=cluster1 2023-08-26T13:40:01.688 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ eval instance=0 2023-08-26T13:40:01.688 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ instance=0 2023-08-26T13:40:01.688 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ daemon_asok_file cluster1:0 cluster1 2023-08-26T13:40:01.689 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ local local_cluster=cluster1:0 2023-08-26T13:40:01.689 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ local cluster=cluster1 2023-08-26T13:40:01.689 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ local instance 2023-08-26T13:40:01.690 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2023-08-26T13:40:01.690 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ local val=cluster1:0 2023-08-26T13:40:01.691 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ local cluster_var_name=local_cluster 2023-08-26T13:40:01.691 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ local instance_var_name=instance 2023-08-26T13:40:01.691 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ cluster=cluster1 2023-08-26T13:40:01.691 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ instance=0 2023-08-26T13:40:01.692 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ '[' 0 = cluster1:0 ']' 2023-08-26T13:40:01.692 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ eval local_cluster=cluster1 2023-08-26T13:40:01.692 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++++ local_cluster=cluster1 2023-08-26T13:40:01.692 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ eval instance=0 2023-08-26T13:40:01.692 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++++ instance=0 2023-08-26T13:40:01.692 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2023-08-26T13:40:01.705 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-08-26T13:40:01.705 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-08-26T13:40:01.705 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-08-26T13:40:01.705 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2023-08-26T13:40:01.805 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:Invalid command: unused arguments: ['mirror/test'] 2023-08-26T13:40:01.805 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:rbd mirror status : get status for rbd mirror 2023-08-26T13:40:01.805 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:admin_socket: invalid command 2023-08-26T13:40:01.808 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-08-26T13:40:01.808 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/test 2023-08-26T13:40:01.809 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local cluster=cluster1:1 2023-08-26T13:40:01.809 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ shift 2023-08-26T13:40:01.809 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local instance 2023-08-26T13:40:01.809 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ set_cluster_instance cluster1:1 cluster instance 2023-08-26T13:40:01.809 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local val=cluster1:1 2023-08-26T13:40:01.809 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local cluster_var_name=cluster 2023-08-26T13:40:01.810 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local instance_var_name=instance 2023-08-26T13:40:01.810 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ cluster=cluster1 2023-08-26T13:40:01.810 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ instance=1 2023-08-26T13:40:01.810 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ '[' 1 = cluster1:1 ']' 2023-08-26T13:40:01.810 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ eval cluster=cluster1 2023-08-26T13:40:01.811 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ cluster=cluster1 2023-08-26T13:40:01.811 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ eval instance=1 2023-08-26T13:40:01.811 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ instance=1 2023-08-26T13:40:01.811 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ daemon_asok_file cluster1:1 cluster1 2023-08-26T13:40:01.811 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ local local_cluster=cluster1:1 2023-08-26T13:40:01.811 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ local cluster=cluster1 2023-08-26T13:40:01.812 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ local instance 2023-08-26T13:40:01.812 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2023-08-26T13:40:01.812 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ local val=cluster1:1 2023-08-26T13:40:01.812 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ local cluster_var_name=local_cluster 2023-08-26T13:40:01.812 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ local instance_var_name=instance 2023-08-26T13:40:01.813 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ cluster=cluster1 2023-08-26T13:40:01.813 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ instance=1 2023-08-26T13:40:01.813 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ '[' 1 = cluster1:1 ']' 2023-08-26T13:40:01.813 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ eval local_cluster=cluster1 2023-08-26T13:40:01.813 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++++ local_cluster=cluster1 2023-08-26T13:40:01.813 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ eval instance=1 2023-08-26T13:40:01.814 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++++ instance=1 2023-08-26T13:40:01.814 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2023-08-26T13:40:01.827 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-08-26T13:40:01.827 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-08-26T13:40:01.827 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-08-26T13:40:01.827 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/test 2023-08-26T13:40:01.921 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:Invalid command: unused arguments: ['mirror/test'] 2023-08-26T13:40:01.921 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:rbd mirror status : get status for rbd mirror 2023-08-26T13:40:01.921 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:admin_socket: invalid command 2023-08-26T13:40:01.924 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-08-26T13:40:01.924 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/test 2023-08-26T13:40:01.924 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local cluster=cluster1:2 2023-08-26T13:40:01.924 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ shift 2023-08-26T13:40:01.925 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local instance 2023-08-26T13:40:01.925 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ set_cluster_instance cluster1:2 cluster instance 2023-08-26T13:40:01.925 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local val=cluster1:2 2023-08-26T13:40:01.925 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local cluster_var_name=cluster 2023-08-26T13:40:01.925 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local instance_var_name=instance 2023-08-26T13:40:01.926 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ cluster=cluster1 2023-08-26T13:40:01.926 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ instance=2 2023-08-26T13:40:01.926 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ '[' 2 = cluster1:2 ']' 2023-08-26T13:40:01.926 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ eval cluster=cluster1 2023-08-26T13:40:01.927 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ cluster=cluster1 2023-08-26T13:40:01.928 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ eval instance=2 2023-08-26T13:40:01.928 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ instance=2 2023-08-26T13:40:01.928 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ daemon_asok_file cluster1:2 cluster1 2023-08-26T13:40:01.928 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ local local_cluster=cluster1:2 2023-08-26T13:40:01.929 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ local cluster=cluster1 2023-08-26T13:40:01.929 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ local instance 2023-08-26T13:40:01.929 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2023-08-26T13:40:01.929 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ local val=cluster1:2 2023-08-26T13:40:01.929 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ local cluster_var_name=local_cluster 2023-08-26T13:40:01.929 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ local instance_var_name=instance 2023-08-26T13:40:01.930 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ cluster=cluster1 2023-08-26T13:40:01.930 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ instance=2 2023-08-26T13:40:01.930 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ '[' 2 = cluster1:2 ']' 2023-08-26T13:40:01.930 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ eval local_cluster=cluster1 2023-08-26T13:40:01.930 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++++ local_cluster=cluster1 2023-08-26T13:40:01.931 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ eval instance=2 2023-08-26T13:40:01.931 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++++ instance=2 2023-08-26T13:40:01.931 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2023-08-26T13:40:01.942 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-08-26T13:40:01.943 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-08-26T13:40:01.943 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-08-26T13:40:01.943 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/test 2023-08-26T13:40:02.047 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ return 0 2023-08-26T13:40:02.048 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ status_result=' "state": "Replaying"' 2023-08-26T13:40:02.048 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ echo ' "state": "Replaying"' 2023-08-26T13:40:02.048 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ grep -i Replaying 2023-08-26T13:40:02.048 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: "state": "Replaying" 2023-08-26T13:40:02.050 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ current_state=started 2023-08-26T13:40:02.051 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ test started = started 2023-08-26T13:40:02.051 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ return 0 2023-08-26T13:40:02.051 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror test 2023-08-26T13:40:02.051 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local local_cluster=cluster1 2023-08-26T13:40:02.051 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster2 2023-08-26T13:40:02.052 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T13:40:02.052 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=test 2023-08-26T13:40:02.052 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ '[' snapshot = journal ']' 2023-08-26T13:40:02.052 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ '[' snapshot = snapshot ']' 2023-08-26T13:40:02.052 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror test 2023-08-26T13:40:02.053 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local local_cluster=cluster1 2023-08-26T13:40:02.053 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster2 2023-08-26T13:40:02.053 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T13:40:02.053 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=test 2023-08-26T13:40:02.053 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ mkfname cluster2-mirror-test.status 2023-08-26T13:40:02.054 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ echo cluster2-mirror-test.status 2023-08-26T13:40:02.054 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ sed -e 's|[/ ]|_|g' 2023-08-26T13:40:02.054 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local status_log=/tmp/tmp.TfDn6BcQ3N/cluster2-mirror-test.status 2023-08-26T13:40:02.054 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ mkfname cluster1-mirror-test.status 2023-08-26T13:40:02.054 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ echo cluster1-mirror-test.status 2023-08-26T13:40:02.054 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ sed -e 's|[/ ]|_|g' 2023-08-26T13:40:02.055 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local local_status_log=/tmp/tmp.TfDn6BcQ3N/cluster1-mirror-test.status 2023-08-26T13:40:02.056 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ mirror_image_snapshot cluster2 mirror test 2023-08-26T13:40:02.056 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster2 2023-08-26T13:40:02.056 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T13:40:02.056 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=test 2023-08-26T13:40:02.057 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/test 2023-08-26T13:40:03.687 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:Snapshot ID: 33 2023-08-26T13:40:03.703 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ get_newest_mirror_snapshot cluster2 mirror test /tmp/tmp.TfDn6BcQ3N/cluster2-mirror-test.status 2023-08-26T13:40:03.703 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster2 2023-08-26T13:40:03.703 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T13:40:03.703 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=test 2023-08-26T13:40:03.704 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local log=/tmp/tmp.TfDn6BcQ3N/cluster2-mirror-test.status 2023-08-26T13:40:03.704 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rbd --cluster cluster2 snap list --all mirror/test --format xml 2023-08-26T13:40:03.704 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-08-26T13:40:03.771 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ xmlstarlet sel -t -v //snapshot/id 2023-08-26T13:40:03.774 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local snapshot_id=33 2023-08-26T13:40:03.774 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ true 2023-08-26T13:40:03.774 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-08-26T13:40:03.774 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ sleep 0.2 2023-08-26T13:40:03.975 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.TfDn6BcQ3N/cluster1-mirror-test.status 2023-08-26T13:40:03.976 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster1 2023-08-26T13:40:03.976 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T13:40:03.976 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=test 2023-08-26T13:40:03.976 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local log=/tmp/tmp.TfDn6BcQ3N/cluster1-mirror-test.status 2023-08-26T13:40:03.976 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2023-08-26T13:40:03.977 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-08-26T13:40:04.023 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ true 2023-08-26T13:40:04.024 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-08-26T13:40:04.024 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:-:1.1: Document is empty 2023-08-26T13:40:04.025 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr: 2023-08-26T13:40:04.025 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:^ 2023-08-26T13:40:04.026 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local primary_snapshot_id= 2023-08-26T13:40:04.026 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ test 33 = '' 2023-08-26T13:40:04.026 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-08-26T13:40:04.026 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ sleep 0.4 2023-08-26T13:40:04.426 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.TfDn6BcQ3N/cluster1-mirror-test.status 2023-08-26T13:40:04.427 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster1 2023-08-26T13:40:04.427 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T13:40:04.427 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=test 2023-08-26T13:40:04.427 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local log=/tmp/tmp.TfDn6BcQ3N/cluster1-mirror-test.status 2023-08-26T13:40:04.428 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2023-08-26T13:40:04.428 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-08-26T13:40:04.476 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ true 2023-08-26T13:40:04.477 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-08-26T13:40:04.478 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:-:1.1: Document is empty 2023-08-26T13:40:04.478 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr: 2023-08-26T13:40:04.478 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:^ 2023-08-26T13:40:04.479 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local primary_snapshot_id= 2023-08-26T13:40:04.479 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ test 33 = '' 2023-08-26T13:40:04.479 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-08-26T13:40:04.479 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ sleep 0.8 2023-08-26T13:40:05.280 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.TfDn6BcQ3N/cluster1-mirror-test.status 2023-08-26T13:40:05.280 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster1 2023-08-26T13:40:05.280 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T13:40:05.281 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=test 2023-08-26T13:40:05.281 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local log=/tmp/tmp.TfDn6BcQ3N/cluster1-mirror-test.status 2023-08-26T13:40:05.281 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2023-08-26T13:40:05.281 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-08-26T13:40:05.362 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ true 2023-08-26T13:40:05.363 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-08-26T13:40:05.364 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:-:1.1: Document is empty 2023-08-26T13:40:05.364 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr: 2023-08-26T13:40:05.364 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:^ 2023-08-26T13:40:05.365 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local primary_snapshot_id= 2023-08-26T13:40:05.365 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ test 33 = '' 2023-08-26T13:40:05.365 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-08-26T13:40:05.366 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ sleep 1.6 2023-08-26T13:40:06.967 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.TfDn6BcQ3N/cluster1-mirror-test.status 2023-08-26T13:40:06.967 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster1 2023-08-26T13:40:06.967 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T13:40:06.967 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=test 2023-08-26T13:40:06.968 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local log=/tmp/tmp.TfDn6BcQ3N/cluster1-mirror-test.status 2023-08-26T13:40:06.968 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2023-08-26T13:40:06.968 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-08-26T13:40:07.017 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-08-26T13:40:07.019 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local primary_snapshot_id=33 2023-08-26T13:40:07.019 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ test 33 = 33 2023-08-26T13:40:07.019 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ return 0 2023-08-26T13:40:07.020 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ wait_for_snap_present cluster1 mirror test snap8 2023-08-26T13:40:07.020 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster1 2023-08-26T13:40:07.020 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T13:40:07.020 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=test 2023-08-26T13:40:07.020 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local snap_name=snap8 2023-08-26T13:40:07.021 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local s 2023-08-26T13:40:07.021 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2023-08-26T13:40:07.021 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ sleep 1 2023-08-26T13:40:08.020 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rbd --cluster cluster1 info mirror/test@snap8 2023-08-26T13:40:08.068 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:rbd image 'test': 2023-08-26T13:40:08.069 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: size 512 MiB in 128 objects 2023-08-26T13:40:08.069 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: order 22 (4 MiB objects) 2023-08-26T13:40:08.069 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: snapshot_count: 16 2023-08-26T13:40:08.069 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: id: 10ab632cdafc 2023-08-26T13:40:08.069 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: block_name_prefix: rbd_data.10ab632cdafc 2023-08-26T13:40:08.070 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: format: 2 2023-08-26T13:40:08.070 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: features: layering, exclusive-lock, non-primary 2023-08-26T13:40:08.070 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: op_features: 2023-08-26T13:40:08.070 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: flags: 2023-08-26T13:40:08.070 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: create_timestamp: Sat Aug 26 13:33:17 2023 2023-08-26T13:40:08.071 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: access_timestamp: Sat Aug 26 13:39:14 2023 2023-08-26T13:40:08.071 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: modify_timestamp: Sat Aug 26 13:33:17 2023 2023-08-26T13:40:08.071 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: protected: False 2023-08-26T13:40:08.071 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: mirroring state: enabled 2023-08-26T13:40:08.072 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: mirroring mode: snapshot 2023-08-26T13:40:08.072 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: mirroring global id: e2f2554d-659e-4467-b7f8-dddd7ccb0ee9 2023-08-26T13:40:08.072 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: mirroring primary: false 2023-08-26T13:40:08.073 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ return 0 2023-08-26T13:40:08.073 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ '[' -n snap7-clean ']' 2023-08-26T13:40:08.074 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ compare_image_snaps mirror test snap7-clean 2023-08-26T13:40:08.074 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T13:40:08.074 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=test 2023-08-26T13:40:08.074 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local snap_name=snap7-clean 2023-08-26T13:40:08.075 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local ret=0 2023-08-26T13:40:08.075 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local rmt_export=/tmp/tmp.TfDn6BcQ3N/cluster2-mirror-test.export 2023-08-26T13:40:08.075 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local loc_export=/tmp/tmp.TfDn6BcQ3N/cluster1-mirror-test.export 2023-08-26T13:40:08.075 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rm -f /tmp/tmp.TfDn6BcQ3N/cluster2-mirror-test.export /tmp/tmp.TfDn6BcQ3N/cluster1-mirror-test.export 2023-08-26T13:40:08.076 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rbd --cluster cluster2 -p mirror export test@snap7-clean /tmp/tmp.TfDn6BcQ3N/cluster2-mirror-test.export 2023-08-26T13:40:08.687 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2023-08-26T13:40:08.696 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rbd --cluster cluster1 -p mirror export test@snap7-clean /tmp/tmp.TfDn6BcQ3N/cluster1-mirror-test.export 2023-08-26T13:40:09.377 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2023-08-26T13:40:09.384 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ cmp /tmp/tmp.TfDn6BcQ3N/cluster2-mirror-test.export /tmp/tmp.TfDn6BcQ3N/cluster1-mirror-test.export 2023-08-26T13:40:09.597 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rm -f /tmp/tmp.TfDn6BcQ3N/cluster2-mirror-test.export /tmp/tmp.TfDn6BcQ3N/cluster1-mirror-test.export 2023-08-26T13:40:09.740 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ return 0 2023-08-26T13:40:09.741 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ compare_image_snaps mirror test snap8 2023-08-26T13:40:09.741 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T13:40:09.741 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=test 2023-08-26T13:40:09.741 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local snap_name=snap8 2023-08-26T13:40:09.742 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local ret=0 2023-08-26T13:40:09.742 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local rmt_export=/tmp/tmp.TfDn6BcQ3N/cluster2-mirror-test.export 2023-08-26T13:40:09.742 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local loc_export=/tmp/tmp.TfDn6BcQ3N/cluster1-mirror-test.export 2023-08-26T13:40:09.742 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rm -f /tmp/tmp.TfDn6BcQ3N/cluster2-mirror-test.export /tmp/tmp.TfDn6BcQ3N/cluster1-mirror-test.export 2023-08-26T13:40:09.742 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rbd --cluster cluster2 -p mirror export test@snap8 /tmp/tmp.TfDn6BcQ3N/cluster2-mirror-test.export 2023-08-26T13:40:10.351 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2023-08-26T13:40:10.359 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rbd --cluster cluster1 -p mirror export test@snap8 /tmp/tmp.TfDn6BcQ3N/cluster1-mirror-test.export 2023-08-26T13:40:11.080 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2023-08-26T13:40:11.087 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ cmp /tmp/tmp.TfDn6BcQ3N/cluster2-mirror-test.export /tmp/tmp.TfDn6BcQ3N/cluster1-mirror-test.export 2023-08-26T13:40:11.299 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rm -f /tmp/tmp.TfDn6BcQ3N/cluster2-mirror-test.export /tmp/tmp.TfDn6BcQ3N/cluster1-mirror-test.export 2023-08-26T13:40:11.441 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ return 0 2023-08-26T13:40:11.442 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ clean_snap_name=snap8-clean 2023-08-26T13:40:11.442 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ create_snap cluster2 mirror test snap8-clean 2023-08-26T13:40:11.442 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster2 2023-08-26T13:40:11.442 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T13:40:11.442 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=test 2023-08-26T13:40:11.443 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local snap_name=snap8-clean 2023-08-26T13:40:11.443 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rbd --cluster cluster2 -p mirror snap create test@snap8-clean --debug-rbd=20 --debug-journaler=20 2023-08-26T13:40:11.997 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ for i in `seq 1 10` 2023-08-26T13:40:11.997 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ stress_write_image cluster2 mirror test 2023-08-26T13:40:11.997 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster2 2023-08-26T13:40:11.997 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T13:40:11.997 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=test 2023-08-26T13:40:11.998 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2023-08-26T13:40:11.998 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local duration=8 2023-08-26T13:40:11.999 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ set +e 2023-08-26T13:40:11.999 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ timeout 8s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror test --debug-rbd=20 --debug-journaler=20 2023-08-26T13:40:12.239 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:100 IOs, 40097792 bytes 2023-08-26T13:40:12.723 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:200 IOs, 135811072 bytes 2023-08-26T13:40:12.937 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:300 IOs, 202597376 bytes 2023-08-26T13:40:13.074 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:400 IOs, 228473856 bytes 2023-08-26T13:40:13.253 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:500 IOs, 271699968 bytes 2023-08-26T13:40:13.429 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:600 IOs, 320410624 bytes 2023-08-26T13:40:13.588 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:700 IOs, 354623488 bytes 2023-08-26T13:40:13.738 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:800 IOs, 383578112 bytes 2023-08-26T13:40:13.850 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:900 IOs, 400824320 bytes 2023-08-26T13:40:14.023 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:1000 IOs, 447641600 bytes 2023-08-26T13:40:14.218 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:1100 IOs, 505402368 bytes 2023-08-26T13:40:14.450 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:1200 IOs, 583147520 bytes 2023-08-26T13:40:14.621 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:1300 IOs, 626662400 bytes 2023-08-26T13:40:14.716 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:1400 IOs, 633014272 bytes 2023-08-26T13:40:14.876 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:1500 IOs, 688663552 bytes 2023-08-26T13:40:15.103 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:1600 IOs, 747949056 bytes 2023-08-26T13:40:15.260 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:1700 IOs, 787775488 bytes 2023-08-26T13:40:15.465 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:1800 IOs, 850725888 bytes 2023-08-26T13:40:15.584 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:1900 IOs, 872224768 bytes 2023-08-26T13:40:15.878 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:2000 IOs, 983169024 bytes 2023-08-26T13:40:16.032 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:2100 IOs, 1018267648 bytes 2023-08-26T13:40:16.237 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:2200 IOs, 1107875840 bytes 2023-08-26T13:40:16.376 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:2300 IOs, 1116066816 bytes 2023-08-26T13:40:16.612 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:2400 IOs, 1192137728 bytes 2023-08-26T13:40:16.739 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:2500 IOs, 1240568832 bytes 2023-08-26T13:40:17.000 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:2600 IOs, 1306787840 bytes 2023-08-26T13:40:17.206 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:2700 IOs, 1357059072 bytes 2023-08-26T13:40:17.409 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:2800 IOs, 1419208704 bytes 2023-08-26T13:40:17.705 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:2900 IOs, 1522740224 bytes 2023-08-26T13:40:17.895 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:3000 IOs, 1580998656 bytes 2023-08-26T13:40:18.089 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:3100 IOs, 1641651200 bytes 2023-08-26T13:40:18.377 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:3200 IOs, 1754274816 bytes 2023-08-26T13:40:18.559 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:3300 IOs, 1806019584 bytes 2023-08-26T13:40:18.702 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:3400 IOs, 1850473472 bytes 2023-08-26T13:40:18.872 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:3500 IOs, 1894114304 bytes 2023-08-26T13:40:19.067 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:3600 IOs, 1955775488 bytes 2023-08-26T13:40:19.164 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:3700 IOs, 1956380672 bytes 2023-08-26T13:40:19.366 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:3800 IOs, 2016807936 bytes 2023-08-26T13:40:19.535 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:3900 IOs, 2073091072 bytes 2023-08-26T13:40:19.685 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:4000 IOs, 2126547968 bytes 2023-08-26T13:40:19.864 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:4100 IOs, 2142387200 bytes 2023-08-26T13:40:19.967 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:4200 IOs, 2154171392 bytes 2023-08-26T13:40:20.003 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ error_code=124 2023-08-26T13:40:20.003 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ set -e 2023-08-26T13:40:20.003 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ '[' 124 -eq 124 ']' 2023-08-26T13:40:20.004 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ return 0 2023-08-26T13:40:20.004 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ wait_for_status_in_pool_dir cluster1 mirror test up+replaying 2023-08-26T13:40:20.004 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster1 2023-08-26T13:40:20.004 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T13:40:20.004 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=test 2023-08-26T13:40:20.005 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local state_pattern=up+replaying 2023-08-26T13:40:20.005 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local description_pattern= 2023-08-26T13:40:20.005 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local service_pattern= 2023-08-26T13:40:20.005 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-08-26T13:40:20.005 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ sleep 1 2023-08-26T13:40:21.004 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ test_status_in_pool_dir cluster1 mirror test up+replaying '' '' 2023-08-26T13:40:21.004 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster1 2023-08-26T13:40:21.005 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T13:40:21.005 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=test 2023-08-26T13:40:21.005 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local state_pattern=up+replaying 2023-08-26T13:40:21.005 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local description_pattern= 2023-08-26T13:40:21.005 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local service_pattern= 2023-08-26T13:40:21.005 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ mkfname cluster1-mirror-test.mirror_status 2023-08-26T13:40:21.006 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ echo cluster1-mirror-test.mirror_status 2023-08-26T13:40:21.006 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ sed -e 's|[/ ]|_|g' 2023-08-26T13:40:21.006 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local status_log=/tmp/tmp.TfDn6BcQ3N/cluster1-mirror-test.mirror_status 2023-08-26T13:40:21.007 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ CEPH_ARGS= 2023-08-26T13:40:21.007 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rbd --cluster cluster1 mirror image status mirror/test 2023-08-26T13:40:21.007 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ tee /tmp/tmp.TfDn6BcQ3N/cluster1-mirror-test.mirror_status 2023-08-26T13:40:21.061 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:test: 2023-08-26T13:40:21.061 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr: global_id: e2f2554d-659e-4467-b7f8-dddd7ccb0ee9 2023-08-26T13:40:21.062 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr: state: up+replaying 2023-08-26T13:40:21.062 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr: description: replaying, {"bytes_per_second":17701478.4,"bytes_per_snapshot":531507712.0,"last_snapshot_bytes":531044352,"last_snapshot_sync_seconds":1,"local_snapshot_timestamp":1693057202,"remote_snapshot_timestamp":1693057202,"replay_state":"idle"} 2023-08-26T13:40:21.062 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr: service: mirror.2 on smithi124 2023-08-26T13:40:21.062 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr: last_update: 2023-08-26 13:40:15 2023-08-26T13:40:21.063 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr: peer_sites: 2023-08-26T13:40:21.063 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr: name: cluster2 2023-08-26T13:40:21.063 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr: state: up+stopped 2023-08-26T13:40:21.063 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr: description: local image is primary 2023-08-26T13:40:21.063 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr: last_update: 2023-08-26 13:40:15 2023-08-26T13:40:21.067 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ grep '^ state: .*up+replaying' /tmp/tmp.TfDn6BcQ3N/cluster1-mirror-test.mirror_status 2023-08-26T13:40:21.068 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: state: up+replaying 2023-08-26T13:40:21.068 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ grep '^ description: .*' /tmp/tmp.TfDn6BcQ3N/cluster1-mirror-test.mirror_status 2023-08-26T13:40:21.069 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: description: replaying, {"bytes_per_second":17701478.4,"bytes_per_snapshot":531507712.0,"last_snapshot_bytes":531044352,"last_snapshot_sync_seconds":1,"local_snapshot_timestamp":1693057202,"remote_snapshot_timestamp":1693057202,"replay_state":"idle"} 2023-08-26T13:40:21.069 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ '[' -n '' ']' 2023-08-26T13:40:21.070 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ echo up+replaying 2023-08-26T13:40:21.070 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ grep '^up+' 2023-08-26T13:40:21.070 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:up+replaying 2023-08-26T13:40:21.071 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ grep 'service: *mirror..* on ' /tmp/tmp.TfDn6BcQ3N/cluster1-mirror-test.mirror_status 2023-08-26T13:40:21.071 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: service: mirror.2 on smithi124 2023-08-26T13:40:21.072 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' /tmp/tmp.TfDn6BcQ3N/cluster1-mirror-test.mirror_status 2023-08-26T13:40:21.074 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local 'last_update=2023-08-26 13:40:15' 2023-08-26T13:40:21.074 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ test_mirror_pool_status_verbose cluster1 mirror test up+replaying '2023-08-26 13:40:15' 2023-08-26T13:40:21.074 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster1 2023-08-26T13:40:21.074 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T13:40:21.074 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=test 2023-08-26T13:40:21.075 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local state_pattern=up+replaying 2023-08-26T13:40:21.075 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local 'prev_last_update=2023-08-26 13:40:15' 2023-08-26T13:40:21.075 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ mkfname cluster1-mirror.mirror_status 2023-08-26T13:40:21.075 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ echo cluster1-mirror.mirror_status 2023-08-26T13:40:21.075 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ sed -e 's|[/ ]|_|g' 2023-08-26T13:40:21.076 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local status_log=/tmp/tmp.TfDn6BcQ3N/cluster1-mirror.mirror_status 2023-08-26T13:40:21.076 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2023-08-26T13:40:21.114 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:rbd: failed to query services: (13) Permission denied 2023-08-26T13:40:21.139 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local last_update state 2023-08-26T13:40:21.140 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update' 2023-08-26T13:40:21.141 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ last_update='2023-08-26 13:40:15' 2023-08-26T13:40:21.142 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state' 2023-08-26T13:40:21.143 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ state=up+replaying 2023-08-26T13:40:21.144 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ echo up+replaying 2023-08-26T13:40:21.144 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ grep up+replaying 2023-08-26T13:40:21.145 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:up+replaying 2023-08-26T13:40:21.145 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ return 0 2023-08-26T13:40:21.146 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ return 0 2023-08-26T13:40:21.146 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ snap_name=snap9 2023-08-26T13:40:21.146 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ create_snap cluster2 mirror test snap9 2023-08-26T13:40:21.146 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster2 2023-08-26T13:40:21.146 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T13:40:21.147 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=test 2023-08-26T13:40:21.147 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local snap_name=snap9 2023-08-26T13:40:21.147 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rbd --cluster cluster2 -p mirror snap create test@snap9 --debug-rbd=20 --debug-journaler=20 2023-08-26T13:40:48.800 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ wait_for_image_replay_started cluster1 mirror test 2023-08-26T13:40:48.801 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster1 2023-08-26T13:40:48.801 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T13:40:48.801 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=test 2023-08-26T13:40:48.801 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ wait_for_image_replay_state cluster1 mirror test started 2023-08-26T13:40:48.801 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster1 2023-08-26T13:40:48.802 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T13:40:48.802 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=test 2023-08-26T13:40:48.802 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local state=started 2023-08-26T13:40:48.802 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local s 2023-08-26T13:40:48.802 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-08-26T13:40:48.803 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ sleep 1 2023-08-26T13:40:49.802 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ test_image_replay_state cluster1 mirror test started 2023-08-26T13:40:49.802 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster1 2023-08-26T13:40:49.803 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T13:40:49.803 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=test 2023-08-26T13:40:49.803 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local test_state=started 2023-08-26T13:40:49.803 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local status_result 2023-08-26T13:40:49.804 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local current_state=stopped 2023-08-26T13:40:49.804 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ admin_daemons cluster1 rbd mirror status mirror/test 2023-08-26T13:40:49.804 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local cluster_instance=cluster1 2023-08-26T13:40:49.804 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ shift 2023-08-26T13:40:49.805 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local cluster=cluster1 2023-08-26T13:40:49.805 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ grep -i state 2023-08-26T13:40:49.805 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local instance=cluster1 2023-08-26T13:40:49.805 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local loop_instance 2023-08-26T13:40:49.805 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-08-26T13:40:49.806 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ sleep 0 2023-08-26T13:40:49.809 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ '[' cluster1 '!=' cluster1 ']' 2023-08-26T13:40:49.809 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ seq 0 3 2023-08-26T13:40:49.809 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-08-26T13:40:49.809 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test 2023-08-26T13:40:49.809 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local cluster=cluster1:0 2023-08-26T13:40:49.810 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ shift 2023-08-26T13:40:49.810 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local instance 2023-08-26T13:40:49.810 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ set_cluster_instance cluster1:0 cluster instance 2023-08-26T13:40:49.810 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local val=cluster1:0 2023-08-26T13:40:49.811 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local cluster_var_name=cluster 2023-08-26T13:40:49.811 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local instance_var_name=instance 2023-08-26T13:40:49.811 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ cluster=cluster1 2023-08-26T13:40:49.811 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ instance=0 2023-08-26T13:40:49.811 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ '[' 0 = cluster1:0 ']' 2023-08-26T13:40:49.812 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ eval cluster=cluster1 2023-08-26T13:40:49.812 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ cluster=cluster1 2023-08-26T13:40:49.812 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ eval instance=0 2023-08-26T13:40:49.813 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ instance=0 2023-08-26T13:40:49.813 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ daemon_asok_file cluster1:0 cluster1 2023-08-26T13:40:49.813 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ local local_cluster=cluster1:0 2023-08-26T13:40:49.813 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ local cluster=cluster1 2023-08-26T13:40:49.814 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ local instance 2023-08-26T13:40:49.814 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2023-08-26T13:40:49.814 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ local val=cluster1:0 2023-08-26T13:40:49.814 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ local cluster_var_name=local_cluster 2023-08-26T13:40:49.814 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ local instance_var_name=instance 2023-08-26T13:40:49.814 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ cluster=cluster1 2023-08-26T13:40:49.815 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ instance=0 2023-08-26T13:40:49.815 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ '[' 0 = cluster1:0 ']' 2023-08-26T13:40:49.815 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ eval local_cluster=cluster1 2023-08-26T13:40:49.815 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++++ local_cluster=cluster1 2023-08-26T13:40:49.815 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ eval instance=0 2023-08-26T13:40:49.815 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++++ instance=0 2023-08-26T13:40:49.816 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2023-08-26T13:40:49.828 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-08-26T13:40:49.829 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-08-26T13:40:49.829 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-08-26T13:40:49.829 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2023-08-26T13:40:49.935 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:Invalid command: unused arguments: ['mirror/test'] 2023-08-26T13:40:49.935 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:rbd mirror status : get status for rbd mirror 2023-08-26T13:40:49.935 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:admin_socket: invalid command 2023-08-26T13:40:49.938 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-08-26T13:40:49.939 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/test 2023-08-26T13:40:49.939 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local cluster=cluster1:1 2023-08-26T13:40:49.939 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ shift 2023-08-26T13:40:49.939 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local instance 2023-08-26T13:40:49.939 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ set_cluster_instance cluster1:1 cluster instance 2023-08-26T13:40:49.940 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local val=cluster1:1 2023-08-26T13:40:49.940 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local cluster_var_name=cluster 2023-08-26T13:40:49.940 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local instance_var_name=instance 2023-08-26T13:40:49.940 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ cluster=cluster1 2023-08-26T13:40:49.940 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ instance=1 2023-08-26T13:40:49.941 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ '[' 1 = cluster1:1 ']' 2023-08-26T13:40:49.941 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ eval cluster=cluster1 2023-08-26T13:40:49.941 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ cluster=cluster1 2023-08-26T13:40:49.941 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ eval instance=1 2023-08-26T13:40:49.941 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ instance=1 2023-08-26T13:40:49.942 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ daemon_asok_file cluster1:1 cluster1 2023-08-26T13:40:49.942 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ local local_cluster=cluster1:1 2023-08-26T13:40:49.942 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ local cluster=cluster1 2023-08-26T13:40:49.942 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ local instance 2023-08-26T13:40:49.942 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2023-08-26T13:40:49.943 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ local val=cluster1:1 2023-08-26T13:40:50.039 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ local cluster_var_name=local_cluster 2023-08-26T13:40:50.039 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ local instance_var_name=instance 2023-08-26T13:40:50.040 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ cluster=cluster1 2023-08-26T13:40:50.040 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ instance=1 2023-08-26T13:40:50.040 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ '[' 1 = cluster1:1 ']' 2023-08-26T13:40:50.040 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ eval local_cluster=cluster1 2023-08-26T13:40:50.040 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++++ local_cluster=cluster1 2023-08-26T13:40:50.041 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ eval instance=1 2023-08-26T13:40:50.041 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++++ instance=1 2023-08-26T13:40:50.041 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2023-08-26T13:40:50.042 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-08-26T13:40:50.042 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-08-26T13:40:50.042 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-08-26T13:40:50.043 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/test 2023-08-26T13:40:50.052 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:Invalid command: unused arguments: ['mirror/test'] 2023-08-26T13:40:50.052 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:rbd mirror status : get status for rbd mirror 2023-08-26T13:40:50.053 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:admin_socket: invalid command 2023-08-26T13:40:50.056 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-08-26T13:40:50.056 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/test 2023-08-26T13:40:50.056 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local cluster=cluster1:2 2023-08-26T13:40:50.056 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ shift 2023-08-26T13:40:50.056 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local instance 2023-08-26T13:40:50.057 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ set_cluster_instance cluster1:2 cluster instance 2023-08-26T13:40:50.057 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local val=cluster1:2 2023-08-26T13:40:50.057 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local cluster_var_name=cluster 2023-08-26T13:40:50.057 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local instance_var_name=instance 2023-08-26T13:40:50.057 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ cluster=cluster1 2023-08-26T13:40:50.058 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ instance=2 2023-08-26T13:40:50.058 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ '[' 2 = cluster1:2 ']' 2023-08-26T13:40:50.058 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ eval cluster=cluster1 2023-08-26T13:40:50.058 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ cluster=cluster1 2023-08-26T13:40:50.058 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ eval instance=2 2023-08-26T13:40:50.059 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ instance=2 2023-08-26T13:40:50.059 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ daemon_asok_file cluster1:2 cluster1 2023-08-26T13:40:50.059 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ local local_cluster=cluster1:2 2023-08-26T13:40:50.059 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ local cluster=cluster1 2023-08-26T13:40:50.059 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ local instance 2023-08-26T13:40:50.060 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2023-08-26T13:40:50.060 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ local val=cluster1:2 2023-08-26T13:40:50.060 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ local cluster_var_name=local_cluster 2023-08-26T13:40:50.060 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ local instance_var_name=instance 2023-08-26T13:40:50.060 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ cluster=cluster1 2023-08-26T13:40:50.060 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ instance=2 2023-08-26T13:40:50.061 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ '[' 2 = cluster1:2 ']' 2023-08-26T13:40:50.061 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ eval local_cluster=cluster1 2023-08-26T13:40:50.061 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++++ local_cluster=cluster1 2023-08-26T13:40:50.061 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ eval instance=2 2023-08-26T13:40:50.061 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++++ instance=2 2023-08-26T13:40:50.062 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2023-08-26T13:40:50.074 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-08-26T13:40:50.074 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-08-26T13:40:50.074 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-08-26T13:40:50.075 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/test 2023-08-26T13:40:50.177 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ return 0 2023-08-26T13:40:50.178 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ status_result=' "state": "Replaying"' 2023-08-26T13:40:50.178 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ echo ' "state": "Replaying"' 2023-08-26T13:40:50.179 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ grep -i Replaying 2023-08-26T13:40:50.179 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: "state": "Replaying" 2023-08-26T13:40:50.181 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ current_state=started 2023-08-26T13:40:50.181 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ test started = started 2023-08-26T13:40:50.182 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ return 0 2023-08-26T13:40:50.182 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror test 2023-08-26T13:40:50.182 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local local_cluster=cluster1 2023-08-26T13:40:50.182 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster2 2023-08-26T13:40:50.182 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T13:40:50.183 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=test 2023-08-26T13:40:50.183 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ '[' snapshot = journal ']' 2023-08-26T13:40:50.183 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ '[' snapshot = snapshot ']' 2023-08-26T13:40:50.183 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror test 2023-08-26T13:40:50.183 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local local_cluster=cluster1 2023-08-26T13:40:50.184 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster2 2023-08-26T13:40:50.184 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T13:40:50.184 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=test 2023-08-26T13:40:50.184 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ mkfname cluster2-mirror-test.status 2023-08-26T13:40:50.184 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ echo cluster2-mirror-test.status 2023-08-26T13:40:50.185 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ sed -e 's|[/ ]|_|g' 2023-08-26T13:40:50.185 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local status_log=/tmp/tmp.TfDn6BcQ3N/cluster2-mirror-test.status 2023-08-26T13:40:50.185 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ mkfname cluster1-mirror-test.status 2023-08-26T13:40:50.185 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ echo cluster1-mirror-test.status 2023-08-26T13:40:50.185 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ sed -e 's|[/ ]|_|g' 2023-08-26T13:40:50.186 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local local_status_log=/tmp/tmp.TfDn6BcQ3N/cluster1-mirror-test.status 2023-08-26T13:40:50.186 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ mirror_image_snapshot cluster2 mirror test 2023-08-26T13:40:50.187 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster2 2023-08-26T13:40:50.187 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T13:40:50.187 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=test 2023-08-26T13:40:50.187 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/test 2023-08-26T13:40:51.800 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:Snapshot ID: 37 2023-08-26T13:40:51.817 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ get_newest_mirror_snapshot cluster2 mirror test /tmp/tmp.TfDn6BcQ3N/cluster2-mirror-test.status 2023-08-26T13:40:51.818 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster2 2023-08-26T13:40:51.818 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T13:40:51.818 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=test 2023-08-26T13:40:51.818 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local log=/tmp/tmp.TfDn6BcQ3N/cluster2-mirror-test.status 2023-08-26T13:40:51.818 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rbd --cluster cluster2 snap list --all mirror/test --format xml 2023-08-26T13:40:51.819 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-08-26T13:40:51.889 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ xmlstarlet sel -t -v //snapshot/id 2023-08-26T13:40:51.891 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local snapshot_id=37 2023-08-26T13:40:51.892 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ true 2023-08-26T13:40:51.892 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-08-26T13:40:51.892 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ sleep 0.2 2023-08-26T13:40:52.093 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.TfDn6BcQ3N/cluster1-mirror-test.status 2023-08-26T13:40:52.093 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster1 2023-08-26T13:40:52.093 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T13:40:52.094 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=test 2023-08-26T13:40:52.094 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local log=/tmp/tmp.TfDn6BcQ3N/cluster1-mirror-test.status 2023-08-26T13:40:52.094 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2023-08-26T13:40:52.094 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-08-26T13:40:52.147 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ true 2023-08-26T13:40:52.148 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-08-26T13:40:52.149 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:-:1.1: Document is empty 2023-08-26T13:40:52.149 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr: 2023-08-26T13:40:52.149 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:^ 2023-08-26T13:40:52.150 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local primary_snapshot_id= 2023-08-26T13:40:52.150 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ test 37 = '' 2023-08-26T13:40:52.150 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-08-26T13:40:52.151 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ sleep 0.4 2023-08-26T13:40:52.551 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.TfDn6BcQ3N/cluster1-mirror-test.status 2023-08-26T13:40:52.551 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster1 2023-08-26T13:40:52.551 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T13:40:52.552 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=test 2023-08-26T13:40:52.552 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local log=/tmp/tmp.TfDn6BcQ3N/cluster1-mirror-test.status 2023-08-26T13:40:52.552 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2023-08-26T13:40:52.552 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-08-26T13:40:52.602 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ true 2023-08-26T13:40:52.603 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-08-26T13:40:52.603 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:-:1.1: Document is empty 2023-08-26T13:40:52.604 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr: 2023-08-26T13:40:52.604 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:^ 2023-08-26T13:40:52.605 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local primary_snapshot_id= 2023-08-26T13:40:52.605 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ test 37 = '' 2023-08-26T13:40:52.605 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-08-26T13:40:52.605 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ sleep 0.8 2023-08-26T13:40:53.405 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.TfDn6BcQ3N/cluster1-mirror-test.status 2023-08-26T13:40:53.406 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster1 2023-08-26T13:40:53.406 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T13:40:53.406 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=test 2023-08-26T13:40:53.406 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local log=/tmp/tmp.TfDn6BcQ3N/cluster1-mirror-test.status 2023-08-26T13:40:53.407 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2023-08-26T13:40:53.407 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-08-26T13:40:53.533 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ true 2023-08-26T13:40:53.533 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-08-26T13:40:53.534 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:-:1.1: Document is empty 2023-08-26T13:40:53.534 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr: 2023-08-26T13:40:53.535 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:^ 2023-08-26T13:40:53.535 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local primary_snapshot_id= 2023-08-26T13:40:53.536 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ test 37 = '' 2023-08-26T13:40:53.536 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-08-26T13:40:53.536 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ sleep 1.6 2023-08-26T13:40:55.136 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.TfDn6BcQ3N/cluster1-mirror-test.status 2023-08-26T13:40:55.136 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster1 2023-08-26T13:40:55.137 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T13:40:55.137 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=test 2023-08-26T13:40:55.137 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local log=/tmp/tmp.TfDn6BcQ3N/cluster1-mirror-test.status 2023-08-26T13:40:55.137 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2023-08-26T13:40:55.137 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-08-26T13:40:55.189 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-08-26T13:40:55.191 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local primary_snapshot_id=37 2023-08-26T13:40:55.191 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ test 37 = 37 2023-08-26T13:40:55.192 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ return 0 2023-08-26T13:40:55.192 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ wait_for_snap_present cluster1 mirror test snap9 2023-08-26T13:40:55.192 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster1 2023-08-26T13:40:55.192 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T13:40:55.192 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=test 2023-08-26T13:40:55.193 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local snap_name=snap9 2023-08-26T13:40:55.193 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local s 2023-08-26T13:40:55.193 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2023-08-26T13:40:55.193 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ sleep 1 2023-08-26T13:40:56.192 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rbd --cluster cluster1 info mirror/test@snap9 2023-08-26T13:40:56.241 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:rbd image 'test': 2023-08-26T13:40:56.241 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: size 512 MiB in 128 objects 2023-08-26T13:40:56.241 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: order 22 (4 MiB objects) 2023-08-26T13:40:56.241 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: snapshot_count: 18 2023-08-26T13:40:56.241 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: id: 10ab632cdafc 2023-08-26T13:40:56.242 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: block_name_prefix: rbd_data.10ab632cdafc 2023-08-26T13:40:56.242 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: format: 2 2023-08-26T13:40:56.242 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: features: layering, exclusive-lock, non-primary 2023-08-26T13:40:56.242 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: op_features: 2023-08-26T13:40:56.242 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: flags: 2023-08-26T13:40:56.242 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: create_timestamp: Sat Aug 26 13:33:17 2023 2023-08-26T13:40:56.243 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: access_timestamp: Sat Aug 26 13:39:14 2023 2023-08-26T13:40:56.243 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: modify_timestamp: Sat Aug 26 13:33:17 2023 2023-08-26T13:40:56.243 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: protected: False 2023-08-26T13:40:56.243 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: mirroring state: enabled 2023-08-26T13:40:56.243 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: mirroring mode: snapshot 2023-08-26T13:40:56.244 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: mirroring global id: e2f2554d-659e-4467-b7f8-dddd7ccb0ee9 2023-08-26T13:40:56.244 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: mirroring primary: false 2023-08-26T13:40:56.245 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ return 0 2023-08-26T13:40:56.245 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ '[' -n snap8-clean ']' 2023-08-26T13:40:56.246 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ compare_image_snaps mirror test snap8-clean 2023-08-26T13:40:56.246 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T13:40:56.246 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=test 2023-08-26T13:40:56.246 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local snap_name=snap8-clean 2023-08-26T13:40:56.246 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local ret=0 2023-08-26T13:40:56.247 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local rmt_export=/tmp/tmp.TfDn6BcQ3N/cluster2-mirror-test.export 2023-08-26T13:40:56.247 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local loc_export=/tmp/tmp.TfDn6BcQ3N/cluster1-mirror-test.export 2023-08-26T13:40:56.247 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rm -f /tmp/tmp.TfDn6BcQ3N/cluster2-mirror-test.export /tmp/tmp.TfDn6BcQ3N/cluster1-mirror-test.export 2023-08-26T13:40:56.247 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rbd --cluster cluster2 -p mirror export test@snap8-clean /tmp/tmp.TfDn6BcQ3N/cluster2-mirror-test.export 2023-08-26T13:40:56.479 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.0 with skip (0.35319324486227033) > weight (0.25) 2023-08-26T13:40:56.480 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.1 with skip (0.8320453205095087) > weight (0.25) 2023-08-26T13:40:56.480 INFO:tasks.rbd_mirror_thrash:kill cluster1.client.mirror.2 2023-08-26T13:40:56.480 INFO:tasks.rbd_mirror.cluster1.client.mirror.2:Sent signal 15 2023-08-26T13:40:56.481 INFO:tasks.rbd_mirror_thrash:waiting for 0 secs before reviving daemons 2023-08-26T13:40:56.481 INFO:tasks.rbd_mirror_thrash:waiting for cluster1.client.mirror.2 2023-08-26T13:40:56.481 INFO:teuthology.orchestra.run:waiting for 600 2023-08-26T13:40:56.481 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi124.stderr:2023-08-26T13:40:56.480+0000 7fea0a400700 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.2 (PID: 105992) UID: 1000 2023-08-26T13:40:56.837 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2023-08-26T13:40:56.848 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rbd --cluster cluster1 -p mirror export test@snap8-clean /tmp/tmp.TfDn6BcQ3N/cluster1-mirror-test.export 2023-08-26T13:40:57.570 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2023-08-26T13:40:57.578 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ cmp /tmp/tmp.TfDn6BcQ3N/cluster2-mirror-test.export /tmp/tmp.TfDn6BcQ3N/cluster1-mirror-test.export 2023-08-26T13:40:57.795 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rm -f /tmp/tmp.TfDn6BcQ3N/cluster2-mirror-test.export /tmp/tmp.TfDn6BcQ3N/cluster1-mirror-test.export 2023-08-26T13:40:57.937 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ return 0 2023-08-26T13:40:57.937 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ compare_image_snaps mirror test snap9 2023-08-26T13:40:57.938 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T13:40:57.938 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=test 2023-08-26T13:40:57.938 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local snap_name=snap9 2023-08-26T13:40:57.938 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local ret=0 2023-08-26T13:40:57.938 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local rmt_export=/tmp/tmp.TfDn6BcQ3N/cluster2-mirror-test.export 2023-08-26T13:40:57.939 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local loc_export=/tmp/tmp.TfDn6BcQ3N/cluster1-mirror-test.export 2023-08-26T13:40:57.939 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rm -f /tmp/tmp.TfDn6BcQ3N/cluster2-mirror-test.export /tmp/tmp.TfDn6BcQ3N/cluster1-mirror-test.export 2023-08-26T13:40:57.939 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rbd --cluster cluster2 -p mirror export test@snap9 /tmp/tmp.TfDn6BcQ3N/cluster2-mirror-test.export 2023-08-26T13:40:58.524 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2023-08-26T13:40:58.534 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rbd --cluster cluster1 -p mirror export test@snap9 /tmp/tmp.TfDn6BcQ3N/cluster1-mirror-test.export 2023-08-26T13:40:59.247 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2023-08-26T13:40:59.256 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ cmp /tmp/tmp.TfDn6BcQ3N/cluster2-mirror-test.export /tmp/tmp.TfDn6BcQ3N/cluster1-mirror-test.export 2023-08-26T13:40:59.474 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rm -f /tmp/tmp.TfDn6BcQ3N/cluster2-mirror-test.export /tmp/tmp.TfDn6BcQ3N/cluster1-mirror-test.export 2023-08-26T13:40:59.621 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ return 0 2023-08-26T13:40:59.621 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ clean_snap_name=snap9-clean 2023-08-26T13:40:59.622 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ create_snap cluster2 mirror test snap9-clean 2023-08-26T13:40:59.622 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster2 2023-08-26T13:40:59.622 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T13:40:59.622 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=test 2023-08-26T13:40:59.622 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local snap_name=snap9-clean 2023-08-26T13:40:59.623 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rbd --cluster cluster2 -p mirror snap create test@snap9-clean --debug-rbd=20 --debug-journaler=20 2023-08-26T13:40:59.994 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ for i in `seq 1 10` 2023-08-26T13:40:59.994 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ stress_write_image cluster2 mirror test 2023-08-26T13:40:59.995 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster2 2023-08-26T13:40:59.995 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T13:40:59.995 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=test 2023-08-26T13:40:59.995 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2023-08-26T13:40:59.996 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local duration=11 2023-08-26T13:40:59.996 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ set +e 2023-08-26T13:40:59.996 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ timeout 11s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror test --debug-rbd=20 --debug-journaler=20 2023-08-26T13:41:00.286 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:100 IOs, 59649024 bytes 2023-08-26T13:41:00.427 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:200 IOs, 96553984 bytes 2023-08-26T13:41:00.691 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:300 IOs, 177281024 bytes 2023-08-26T13:41:00.784 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:400 IOs, 177905664 bytes 2023-08-26T13:41:00.872 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:500 IOs, 181234688 bytes 2023-08-26T13:41:00.997 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:600 IOs, 223670272 bytes 2023-08-26T13:41:01.199 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:700 IOs, 268507136 bytes 2023-08-26T13:41:01.335 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:800 IOs, 291467264 bytes 2023-08-26T13:41:01.414 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:900 IOs, 292104192 bytes 2023-08-26T13:41:01.571 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:1000 IOs, 337725440 bytes 2023-08-26T13:41:01.690 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:1100 IOs, 360049664 bytes 2023-08-26T13:41:01.815 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:1200 IOs, 385059840 bytes 2023-08-26T13:41:01.832 INFO:tasks.ceph.cluster2.osd.2.smithi124.stderr:2023-08-26T13:41:01.831+0000 7fb790854700 -1 reset not still connected to 0x5598e1d43110 2023-08-26T13:41:01.997 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:1300 IOs, 435019776 bytes 2023-08-26T13:41:02.174 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:1400 IOs, 481574912 bytes 2023-08-26T13:41:02.376 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:1500 IOs, 573995008 bytes 2023-08-26T13:41:02.583 INFO:tasks.rbd_mirror_thrash:reviving cluster1.client.mirror.2 2023-08-26T13:41:02.583 INFO:tasks.rbd_mirror.cluster1.client.mirror.2:Restarting daemon 2023-08-26T13:41:02.584 DEBUG:teuthology.orchestra.run.smithi124:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.2 2023-08-26T13:41:02.589 INFO:tasks.rbd_mirror.cluster1.client.mirror.2:Started 2023-08-26T13:41:02.589 INFO:tasks.rbd_mirror_thrash:waiting for 80 secs before thrashing 2023-08-26T13:41:02.589 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:1600 IOs, 633455616 bytes 2023-08-26T13:41:02.710 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:1700 IOs, 645731328 bytes 2023-08-26T13:41:02.867 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:1800 IOs, 679629824 bytes 2023-08-26T13:41:02.977 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:1900 IOs, 694515712 bytes 2023-08-26T13:41:03.064 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:2000 IOs, 695166976 bytes 2023-08-26T13:41:03.152 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:2100 IOs, 695781376 bytes 2023-08-26T13:41:03.379 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:2200 IOs, 772229120 bytes 2023-08-26T13:41:03.534 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:2300 IOs, 804734976 bytes 2023-08-26T13:41:03.792 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:2400 IOs, 887590912 bytes 2023-08-26T13:41:03.933 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:2500 IOs, 927753216 bytes 2023-08-26T13:41:04.102 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:2600 IOs, 955309056 bytes 2023-08-26T13:41:04.222 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:2700 IOs, 972830720 bytes 2023-08-26T13:41:04.398 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:2800 IOs, 1015608320 bytes 2023-08-26T13:41:04.596 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:2900 IOs, 1069360128 bytes 2023-08-26T13:41:04.766 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:3000 IOs, 1112375296 bytes 2023-08-26T13:41:04.855 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:3100 IOs, 1112998912 bytes 2023-08-26T13:41:05.063 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:3200 IOs, 1171492864 bytes 2023-08-26T13:41:05.258 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:3300 IOs, 1235900416 bytes 2023-08-26T13:41:05.437 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:3400 IOs, 1308219392 bytes 2023-08-26T13:41:05.645 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:3500 IOs, 1348770816 bytes 2023-08-26T13:41:05.816 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:3600 IOs, 1390590976 bytes 2023-08-26T13:41:06.025 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:3700 IOs, 1452368896 bytes 2023-08-26T13:41:06.269 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:3800 IOs, 1536209920 bytes 2023-08-26T13:41:06.476 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:3900 IOs, 1609193472 bytes 2023-08-26T13:41:06.608 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:4000 IOs, 1619386368 bytes 2023-08-26T13:41:06.745 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:4100 IOs, 1638387712 bytes 2023-08-26T13:41:06.847 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:4200 IOs, 1659110400 bytes 2023-08-26T13:41:07.029 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:4300 IOs, 1688750080 bytes 2023-08-26T13:41:07.168 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:4400 IOs, 1712578560 bytes 2023-08-26T13:41:07.322 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:4500 IOs, 1744911360 bytes 2023-08-26T13:41:07.465 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:4600 IOs, 1770411008 bytes 2023-08-26T13:41:07.671 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:4700 IOs, 1855019008 bytes 2023-08-26T13:41:07.903 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:4800 IOs, 1913315328 bytes 2023-08-26T13:41:08.118 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:4900 IOs, 1982062592 bytes 2023-08-26T13:41:08.287 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:5000 IOs, 2018518016 bytes 2023-08-26T13:41:08.406 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:5100 IOs, 2040249344 bytes 2023-08-26T13:41:08.562 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:5200 IOs, 2072441856 bytes 2023-08-26T13:41:08.725 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:5300 IOs, 2111908864 bytes 2023-08-26T13:41:08.856 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:5400 IOs, 2144188416 bytes 2023-08-26T13:41:08.997 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:5500 IOs, 2167945216 bytes 2023-08-26T13:41:09.102 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:5600 IOs, 2180884480 bytes 2023-08-26T13:41:09.241 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:5700 IOs, 2207662080 bytes 2023-08-26T13:41:09.476 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:5800 IOs, 2278428672 bytes 2023-08-26T13:41:09.720 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:5900 IOs, 2353850368 bytes 2023-08-26T13:41:09.864 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:6000 IOs, 2379671552 bytes 2023-08-26T13:41:10.001 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:6100 IOs, 2406459392 bytes 2023-08-26T13:41:10.179 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:6200 IOs, 2458586112 bytes 2023-08-26T13:41:10.345 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:6300 IOs, 2499221504 bytes 2023-08-26T13:41:10.489 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:6400 IOs, 2531505152 bytes 2023-08-26T13:41:10.615 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:6500 IOs, 2548180992 bytes 2023-08-26T13:41:10.804 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:6600 IOs, 2595232768 bytes 2023-08-26T13:41:10.900 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:6700 IOs, 2595835904 bytes 2023-08-26T13:41:11.001 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ error_code=124 2023-08-26T13:41:11.001 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ set -e 2023-08-26T13:41:11.001 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ '[' 124 -eq 124 ']' 2023-08-26T13:41:11.001 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ return 0 2023-08-26T13:41:11.002 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ wait_for_status_in_pool_dir cluster1 mirror test up+replaying 2023-08-26T13:41:11.002 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster1 2023-08-26T13:41:11.002 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T13:41:11.002 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=test 2023-08-26T13:41:11.002 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local state_pattern=up+replaying 2023-08-26T13:41:11.003 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local description_pattern= 2023-08-26T13:41:11.003 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local service_pattern= 2023-08-26T13:41:11.003 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-08-26T13:41:11.003 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ sleep 1 2023-08-26T13:41:12.002 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ test_status_in_pool_dir cluster1 mirror test up+replaying '' '' 2023-08-26T13:41:12.002 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster1 2023-08-26T13:41:12.003 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T13:41:12.003 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=test 2023-08-26T13:41:12.003 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local state_pattern=up+replaying 2023-08-26T13:41:12.003 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local description_pattern= 2023-08-26T13:41:12.003 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local service_pattern= 2023-08-26T13:41:12.004 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ mkfname cluster1-mirror-test.mirror_status 2023-08-26T13:41:12.004 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ echo cluster1-mirror-test.mirror_status 2023-08-26T13:41:12.004 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ sed -e 's|[/ ]|_|g' 2023-08-26T13:41:12.005 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local status_log=/tmp/tmp.TfDn6BcQ3N/cluster1-mirror-test.mirror_status 2023-08-26T13:41:12.005 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ CEPH_ARGS= 2023-08-26T13:41:12.005 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rbd --cluster cluster1 mirror image status mirror/test 2023-08-26T13:41:12.005 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ tee /tmp/tmp.TfDn6BcQ3N/cluster1-mirror-test.mirror_status 2023-08-26T13:41:12.058 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:test: 2023-08-26T13:41:12.058 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr: global_id: e2f2554d-659e-4467-b7f8-dddd7ccb0ee9 2023-08-26T13:41:12.059 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr: state: down+unknown 2023-08-26T13:41:12.059 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr: description: status not found 2023-08-26T13:41:12.059 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr: last_update: 2023-08-26T13:41:12.059 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr: peer_sites: 2023-08-26T13:41:12.059 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr: name: cluster2 2023-08-26T13:41:12.060 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr: state: up+stopped 2023-08-26T13:41:12.060 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr: description: local image is primary 2023-08-26T13:41:12.060 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr: last_update: 2023-08-26 13:40:45 2023-08-26T13:41:12.063 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ grep '^ state: .*up+replaying' /tmp/tmp.TfDn6BcQ3N/cluster1-mirror-test.mirror_status 2023-08-26T13:41:12.064 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ return 1 2023-08-26T13:41:12.064 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-08-26T13:41:12.064 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ sleep 2 2023-08-26T13:41:14.066 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ test_status_in_pool_dir cluster1 mirror test up+replaying '' '' 2023-08-26T13:41:14.066 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster1 2023-08-26T13:41:14.066 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T13:41:14.067 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=test 2023-08-26T13:41:14.067 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local state_pattern=up+replaying 2023-08-26T13:41:14.067 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local description_pattern= 2023-08-26T13:41:14.067 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local service_pattern= 2023-08-26T13:41:14.068 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ mkfname cluster1-mirror-test.mirror_status 2023-08-26T13:41:14.068 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ echo cluster1-mirror-test.mirror_status 2023-08-26T13:41:14.068 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ sed -e 's|[/ ]|_|g' 2023-08-26T13:41:14.069 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local status_log=/tmp/tmp.TfDn6BcQ3N/cluster1-mirror-test.mirror_status 2023-08-26T13:41:14.069 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ CEPH_ARGS= 2023-08-26T13:41:14.069 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rbd --cluster cluster1 mirror image status mirror/test 2023-08-26T13:41:14.070 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ tee /tmp/tmp.TfDn6BcQ3N/cluster1-mirror-test.mirror_status 2023-08-26T13:41:14.118 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:test: 2023-08-26T13:41:14.119 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr: global_id: e2f2554d-659e-4467-b7f8-dddd7ccb0ee9 2023-08-26T13:41:14.119 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr: state: down+unknown 2023-08-26T13:41:14.119 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr: description: status not found 2023-08-26T13:41:14.119 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr: last_update: 2023-08-26T13:41:14.120 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr: peer_sites: 2023-08-26T13:41:14.120 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr: name: cluster2 2023-08-26T13:41:14.120 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr: state: up+stopped 2023-08-26T13:41:14.121 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr: description: local image is primary 2023-08-26T13:41:14.121 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr: last_update: 2023-08-26 13:40:45 2023-08-26T13:41:14.123 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ grep '^ state: .*up+replaying' /tmp/tmp.TfDn6BcQ3N/cluster1-mirror-test.mirror_status 2023-08-26T13:41:14.124 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ return 1 2023-08-26T13:41:14.124 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-08-26T13:41:14.124 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ sleep 4 2023-08-26T13:41:18.126 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ test_status_in_pool_dir cluster1 mirror test up+replaying '' '' 2023-08-26T13:41:18.126 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster1 2023-08-26T13:41:18.126 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T13:41:18.126 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=test 2023-08-26T13:41:18.127 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local state_pattern=up+replaying 2023-08-26T13:41:18.127 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local description_pattern= 2023-08-26T13:41:18.127 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local service_pattern= 2023-08-26T13:41:18.127 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ mkfname cluster1-mirror-test.mirror_status 2023-08-26T13:41:18.128 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ echo cluster1-mirror-test.mirror_status 2023-08-26T13:41:18.128 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ sed -e 's|[/ ]|_|g' 2023-08-26T13:41:18.128 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local status_log=/tmp/tmp.TfDn6BcQ3N/cluster1-mirror-test.mirror_status 2023-08-26T13:41:18.128 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ CEPH_ARGS= 2023-08-26T13:41:18.129 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rbd --cluster cluster1 mirror image status mirror/test 2023-08-26T13:41:18.129 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ tee /tmp/tmp.TfDn6BcQ3N/cluster1-mirror-test.mirror_status 2023-08-26T13:41:18.178 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:test: 2023-08-26T13:41:18.178 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr: global_id: e2f2554d-659e-4467-b7f8-dddd7ccb0ee9 2023-08-26T13:41:18.178 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr: state: down+unknown 2023-08-26T13:41:18.179 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr: description: status not found 2023-08-26T13:41:18.179 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr: last_update: 2023-08-26T13:41:18.179 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr: peer_sites: 2023-08-26T13:41:18.179 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr: name: cluster2 2023-08-26T13:41:18.179 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr: state: up+stopped 2023-08-26T13:41:18.180 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr: description: local image is primary 2023-08-26T13:41:18.180 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr: last_update: 2023-08-26 13:41:15 2023-08-26T13:41:18.183 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ grep '^ state: .*up+replaying' /tmp/tmp.TfDn6BcQ3N/cluster1-mirror-test.mirror_status 2023-08-26T13:41:18.184 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ return 1 2023-08-26T13:41:18.185 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-08-26T13:41:18.185 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ sleep 8 2023-08-26T13:41:19.477 INFO:tasks.rbd_mirror_thrash:kill cluster2.client.mirror.0 2023-08-26T13:41:19.478 INFO:tasks.rbd_mirror.cluster2.client.mirror.0:Sent signal 15 2023-08-26T13:41:19.478 INFO:tasks.rbd_mirror_thrash:waiting for 8 secs before reviving daemons 2023-08-26T13:41:19.479 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi124.stderr:2023-08-26T13:41:19.477+0000 7f62d3a0c700 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.0 (PID: 108073) UID: 1000 2023-08-26T13:41:26.186 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ test_status_in_pool_dir cluster1 mirror test up+replaying '' '' 2023-08-26T13:41:26.186 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster1 2023-08-26T13:41:26.186 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T13:41:26.187 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=test 2023-08-26T13:41:26.187 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local state_pattern=up+replaying 2023-08-26T13:41:26.187 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local description_pattern= 2023-08-26T13:41:26.187 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local service_pattern= 2023-08-26T13:41:26.187 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ mkfname cluster1-mirror-test.mirror_status 2023-08-26T13:41:26.188 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ echo cluster1-mirror-test.mirror_status 2023-08-26T13:41:26.188 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ sed -e 's|[/ ]|_|g' 2023-08-26T13:41:26.189 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local status_log=/tmp/tmp.TfDn6BcQ3N/cluster1-mirror-test.mirror_status 2023-08-26T13:41:26.189 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ CEPH_ARGS= 2023-08-26T13:41:26.189 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rbd --cluster cluster1 mirror image status mirror/test 2023-08-26T13:41:26.189 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ tee /tmp/tmp.TfDn6BcQ3N/cluster1-mirror-test.mirror_status 2023-08-26T13:41:26.241 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:test: 2023-08-26T13:41:26.241 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr: global_id: e2f2554d-659e-4467-b7f8-dddd7ccb0ee9 2023-08-26T13:41:26.241 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr: state: up+replaying 2023-08-26T13:41:26.241 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr: description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":0.0,"last_snapshot_bytes":0,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1693057250,"remote_snapshot_timestamp":1693057250,"replay_state":"idle"} 2023-08-26T13:41:26.242 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr: service: mirror.3 on smithi124 2023-08-26T13:41:26.242 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr: last_update: 2023-08-26 13:41:24 2023-08-26T13:41:26.242 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr: peer_sites: 2023-08-26T13:41:26.242 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr: name: cluster2 2023-08-26T13:41:26.242 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr: state: up+stopped 2023-08-26T13:41:26.243 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr: description: local image is primary 2023-08-26T13:41:26.243 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr: last_update: 2023-08-26 13:41:15 2023-08-26T13:41:26.246 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ grep '^ state: .*up+replaying' /tmp/tmp.TfDn6BcQ3N/cluster1-mirror-test.mirror_status 2023-08-26T13:41:26.247 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: state: up+replaying 2023-08-26T13:41:26.247 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ grep '^ description: .*' /tmp/tmp.TfDn6BcQ3N/cluster1-mirror-test.mirror_status 2023-08-26T13:41:26.248 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":0.0,"last_snapshot_bytes":0,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1693057250,"remote_snapshot_timestamp":1693057250,"replay_state":"idle"} 2023-08-26T13:41:26.248 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ '[' -n '' ']' 2023-08-26T13:41:26.249 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ echo up+replaying 2023-08-26T13:41:26.249 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ grep '^up+' 2023-08-26T13:41:26.249 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:up+replaying 2023-08-26T13:41:26.250 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ grep 'service: *mirror..* on ' /tmp/tmp.TfDn6BcQ3N/cluster1-mirror-test.mirror_status 2023-08-26T13:41:26.250 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: service: mirror.3 on smithi124 2023-08-26T13:41:26.251 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' /tmp/tmp.TfDn6BcQ3N/cluster1-mirror-test.mirror_status 2023-08-26T13:41:26.252 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local 'last_update=2023-08-26 13:41:24' 2023-08-26T13:41:26.253 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ test_mirror_pool_status_verbose cluster1 mirror test up+replaying '2023-08-26 13:41:24' 2023-08-26T13:41:26.253 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster1 2023-08-26T13:41:26.253 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T13:41:26.253 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=test 2023-08-26T13:41:26.254 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local state_pattern=up+replaying 2023-08-26T13:41:26.254 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local 'prev_last_update=2023-08-26 13:41:24' 2023-08-26T13:41:26.254 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ mkfname cluster1-mirror.mirror_status 2023-08-26T13:41:26.254 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ echo cluster1-mirror.mirror_status 2023-08-26T13:41:26.254 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ sed -e 's|[/ ]|_|g' 2023-08-26T13:41:26.255 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local status_log=/tmp/tmp.TfDn6BcQ3N/cluster1-mirror.mirror_status 2023-08-26T13:41:26.255 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2023-08-26T13:41:26.291 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:rbd: failed to query services: (13) Permission denied 2023-08-26T13:41:26.316 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local last_update state 2023-08-26T13:41:26.316 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update' 2023-08-26T13:41:26.318 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ last_update='2023-08-26 13:41:24' 2023-08-26T13:41:26.318 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state' 2023-08-26T13:41:26.320 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ state=up+replaying 2023-08-26T13:41:26.320 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ echo up+replaying 2023-08-26T13:41:26.320 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ grep up+replaying 2023-08-26T13:41:26.321 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:up+replaying 2023-08-26T13:41:26.322 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ return 0 2023-08-26T13:41:26.322 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ return 0 2023-08-26T13:41:26.322 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ snap_name=snap10 2023-08-26T13:41:26.322 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ create_snap cluster2 mirror test snap10 2023-08-26T13:41:26.323 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster2 2023-08-26T13:41:26.323 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T13:41:26.323 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=test 2023-08-26T13:41:26.323 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local snap_name=snap10 2023-08-26T13:41:26.323 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rbd --cluster cluster2 -p mirror snap create test@snap10 --debug-rbd=20 --debug-journaler=20 2023-08-26T13:41:27.479 INFO:tasks.rbd_mirror_thrash:waiting for cluster2.client.mirror.0 2023-08-26T13:41:27.479 INFO:teuthology.orchestra.run:waiting for 600 2023-08-26T13:41:27.480 INFO:tasks.rbd_mirror_thrash:reviving cluster2.client.mirror.0 2023-08-26T13:41:27.480 INFO:tasks.rbd_mirror.cluster2.client.mirror.0:Restarting daemon 2023-08-26T13:41:27.480 DEBUG:teuthology.orchestra.run.smithi124:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.0 2023-08-26T13:41:27.483 INFO:tasks.rbd_mirror.cluster2.client.mirror.0:Started 2023-08-26T13:41:27.483 INFO:tasks.rbd_mirror_thrash:waiting for 95 secs before thrashing 2023-08-26T13:41:42.018 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ wait_for_image_replay_started cluster1 mirror test 2023-08-26T13:41:42.019 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster1 2023-08-26T13:41:42.019 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T13:41:42.019 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=test 2023-08-26T13:41:42.019 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ wait_for_image_replay_state cluster1 mirror test started 2023-08-26T13:41:42.020 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster1 2023-08-26T13:41:42.020 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T13:41:42.020 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=test 2023-08-26T13:41:42.020 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local state=started 2023-08-26T13:41:42.020 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local s 2023-08-26T13:41:42.021 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-08-26T13:41:42.021 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ sleep 1 2023-08-26T13:41:43.020 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ test_image_replay_state cluster1 mirror test started 2023-08-26T13:41:43.020 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster1 2023-08-26T13:41:43.020 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T13:41:43.020 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=test 2023-08-26T13:41:43.021 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local test_state=started 2023-08-26T13:41:43.021 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local status_result 2023-08-26T13:41:43.021 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local current_state=stopped 2023-08-26T13:41:43.025 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ admin_daemons cluster1 rbd mirror status mirror/test 2023-08-26T13:41:43.025 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local cluster_instance=cluster1 2023-08-26T13:41:43.025 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ shift 2023-08-26T13:41:43.026 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ grep -i state 2023-08-26T13:41:43.026 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local cluster=cluster1 2023-08-26T13:41:43.026 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local instance=cluster1 2023-08-26T13:41:43.026 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local loop_instance 2023-08-26T13:41:43.026 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-08-26T13:41:43.027 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ sleep 0 2023-08-26T13:41:43.027 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ '[' cluster1 '!=' cluster1 ']' 2023-08-26T13:41:43.027 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ seq 0 3 2023-08-26T13:41:43.027 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-08-26T13:41:43.027 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test 2023-08-26T13:41:43.028 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local cluster=cluster1:0 2023-08-26T13:41:43.028 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ shift 2023-08-26T13:41:43.028 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local instance 2023-08-26T13:41:43.028 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ set_cluster_instance cluster1:0 cluster instance 2023-08-26T13:41:43.028 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local val=cluster1:0 2023-08-26T13:41:43.029 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local cluster_var_name=cluster 2023-08-26T13:41:43.029 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local instance_var_name=instance 2023-08-26T13:41:43.029 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ cluster=cluster1 2023-08-26T13:41:43.029 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ instance=0 2023-08-26T13:41:43.029 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ '[' 0 = cluster1:0 ']' 2023-08-26T13:41:43.029 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ eval cluster=cluster1 2023-08-26T13:41:43.030 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ cluster=cluster1 2023-08-26T13:41:43.030 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ eval instance=0 2023-08-26T13:41:43.030 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ instance=0 2023-08-26T13:41:43.030 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ daemon_asok_file cluster1:0 cluster1 2023-08-26T13:41:43.030 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ local local_cluster=cluster1:0 2023-08-26T13:41:43.031 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ local cluster=cluster1 2023-08-26T13:41:43.031 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ local instance 2023-08-26T13:41:43.031 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2023-08-26T13:41:43.031 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ local val=cluster1:0 2023-08-26T13:41:43.031 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ local cluster_var_name=local_cluster 2023-08-26T13:41:43.031 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ local instance_var_name=instance 2023-08-26T13:41:43.032 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ cluster=cluster1 2023-08-26T13:41:43.032 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ instance=0 2023-08-26T13:41:43.032 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ '[' 0 = cluster1:0 ']' 2023-08-26T13:41:43.032 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ eval local_cluster=cluster1 2023-08-26T13:41:43.032 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++++ local_cluster=cluster1 2023-08-26T13:41:43.033 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ eval instance=0 2023-08-26T13:41:43.033 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++++ instance=0 2023-08-26T13:41:43.033 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2023-08-26T13:41:43.046 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-08-26T13:41:43.046 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-08-26T13:41:43.046 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-08-26T13:41:43.047 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2023-08-26T13:41:43.145 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:Invalid command: unused arguments: ['mirror/test'] 2023-08-26T13:41:43.145 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:rbd mirror status : get status for rbd mirror 2023-08-26T13:41:43.145 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:admin_socket: invalid command 2023-08-26T13:41:43.148 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-08-26T13:41:43.148 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/test 2023-08-26T13:41:43.148 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local cluster=cluster1:1 2023-08-26T13:41:43.149 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ shift 2023-08-26T13:41:43.149 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local instance 2023-08-26T13:41:43.149 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ set_cluster_instance cluster1:1 cluster instance 2023-08-26T13:41:43.149 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local val=cluster1:1 2023-08-26T13:41:43.149 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local cluster_var_name=cluster 2023-08-26T13:41:43.150 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local instance_var_name=instance 2023-08-26T13:41:43.150 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ cluster=cluster1 2023-08-26T13:41:43.150 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ instance=1 2023-08-26T13:41:43.150 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ '[' 1 = cluster1:1 ']' 2023-08-26T13:41:43.150 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ eval cluster=cluster1 2023-08-26T13:41:43.151 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ cluster=cluster1 2023-08-26T13:41:43.151 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ eval instance=1 2023-08-26T13:41:43.151 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ instance=1 2023-08-26T13:41:43.151 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ daemon_asok_file cluster1:1 cluster1 2023-08-26T13:41:43.151 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ local local_cluster=cluster1:1 2023-08-26T13:41:43.152 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ local cluster=cluster1 2023-08-26T13:41:43.152 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ local instance 2023-08-26T13:41:43.152 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2023-08-26T13:41:43.152 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ local val=cluster1:1 2023-08-26T13:41:43.152 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ local cluster_var_name=local_cluster 2023-08-26T13:41:43.153 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ local instance_var_name=instance 2023-08-26T13:41:43.153 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ cluster=cluster1 2023-08-26T13:41:43.153 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ instance=1 2023-08-26T13:41:43.153 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ '[' 1 = cluster1:1 ']' 2023-08-26T13:41:43.153 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ eval local_cluster=cluster1 2023-08-26T13:41:43.153 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++++ local_cluster=cluster1 2023-08-26T13:41:43.154 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ eval instance=1 2023-08-26T13:41:43.154 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++++ instance=1 2023-08-26T13:41:43.154 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2023-08-26T13:41:43.166 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-08-26T13:41:43.167 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-08-26T13:41:43.167 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-08-26T13:41:43.167 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/test 2023-08-26T13:41:43.260 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:Invalid command: unused arguments: ['mirror/test'] 2023-08-26T13:41:43.260 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:rbd mirror status : get status for rbd mirror 2023-08-26T13:41:43.261 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:admin_socket: invalid command 2023-08-26T13:41:43.263 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-08-26T13:41:43.264 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/test 2023-08-26T13:41:43.264 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local cluster=cluster1:2 2023-08-26T13:41:43.264 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ shift 2023-08-26T13:41:43.264 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local instance 2023-08-26T13:41:43.265 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ set_cluster_instance cluster1:2 cluster instance 2023-08-26T13:41:43.265 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local val=cluster1:2 2023-08-26T13:41:43.265 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local cluster_var_name=cluster 2023-08-26T13:41:43.265 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local instance_var_name=instance 2023-08-26T13:41:43.265 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ cluster=cluster1 2023-08-26T13:41:43.265 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ instance=2 2023-08-26T13:41:43.266 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ '[' 2 = cluster1:2 ']' 2023-08-26T13:41:43.266 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ eval cluster=cluster1 2023-08-26T13:41:43.266 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ cluster=cluster1 2023-08-26T13:41:43.266 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ eval instance=2 2023-08-26T13:41:43.266 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ instance=2 2023-08-26T13:41:43.267 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ daemon_asok_file cluster1:2 cluster1 2023-08-26T13:41:43.267 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ local local_cluster=cluster1:2 2023-08-26T13:41:43.267 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ local cluster=cluster1 2023-08-26T13:41:43.267 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ local instance 2023-08-26T13:41:43.267 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2023-08-26T13:41:43.268 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ local val=cluster1:2 2023-08-26T13:41:43.268 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ local cluster_var_name=local_cluster 2023-08-26T13:41:43.268 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ local instance_var_name=instance 2023-08-26T13:41:43.268 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ cluster=cluster1 2023-08-26T13:41:43.268 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ instance=2 2023-08-26T13:41:43.268 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ '[' 2 = cluster1:2 ']' 2023-08-26T13:41:43.269 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ eval local_cluster=cluster1 2023-08-26T13:41:43.269 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++++ local_cluster=cluster1 2023-08-26T13:41:43.269 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ eval instance=2 2023-08-26T13:41:43.269 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++++ instance=2 2023-08-26T13:41:43.269 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2023-08-26T13:41:43.282 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-08-26T13:41:43.282 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-08-26T13:41:43.282 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-08-26T13:41:43.282 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/test 2023-08-26T13:41:43.376 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:Invalid command: unused arguments: ['mirror/test'] 2023-08-26T13:41:43.376 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:rbd mirror status : get status for rbd mirror 2023-08-26T13:41:43.376 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:admin_socket: invalid command 2023-08-26T13:41:43.379 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-08-26T13:41:43.379 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ admin_daemon cluster1:3 rbd mirror status mirror/test 2023-08-26T13:41:43.380 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local cluster=cluster1:3 2023-08-26T13:41:43.380 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ shift 2023-08-26T13:41:43.380 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local instance 2023-08-26T13:41:43.380 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ set_cluster_instance cluster1:3 cluster instance 2023-08-26T13:41:43.380 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local val=cluster1:3 2023-08-26T13:41:43.381 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local cluster_var_name=cluster 2023-08-26T13:41:43.381 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local instance_var_name=instance 2023-08-26T13:41:43.381 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ cluster=cluster1 2023-08-26T13:41:43.381 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ instance=3 2023-08-26T13:41:43.381 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ '[' 3 = cluster1:3 ']' 2023-08-26T13:41:43.382 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ eval cluster=cluster1 2023-08-26T13:41:43.382 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ cluster=cluster1 2023-08-26T13:41:43.382 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ eval instance=3 2023-08-26T13:41:43.382 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ instance=3 2023-08-26T13:41:43.382 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ daemon_asok_file cluster1:3 cluster1 2023-08-26T13:41:43.383 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ local local_cluster=cluster1:3 2023-08-26T13:41:43.383 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ local cluster=cluster1 2023-08-26T13:41:43.383 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ local instance 2023-08-26T13:41:43.383 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ set_cluster_instance cluster1:3 local_cluster instance 2023-08-26T13:41:43.383 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ local val=cluster1:3 2023-08-26T13:41:43.384 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ local cluster_var_name=local_cluster 2023-08-26T13:41:43.384 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ local instance_var_name=instance 2023-08-26T13:41:43.384 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ cluster=cluster1 2023-08-26T13:41:43.384 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ instance=3 2023-08-26T13:41:43.384 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ '[' 3 = cluster1:3 ']' 2023-08-26T13:41:43.384 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ eval local_cluster=cluster1 2023-08-26T13:41:43.385 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++++ local_cluster=cluster1 2023-08-26T13:41:43.385 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ eval instance=3 2023-08-26T13:41:43.385 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++++ instance=3 2023-08-26T13:41:43.385 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2023-08-26T13:41:43.398 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2023-08-26T13:41:43.398 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2023-08-26T13:41:43.398 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2023-08-26T13:41:43.398 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror status mirror/test 2023-08-26T13:41:43.502 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ return 0 2023-08-26T13:41:43.503 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ status_result=' "state": "Replaying"' 2023-08-26T13:41:43.503 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ echo ' "state": "Replaying"' 2023-08-26T13:41:43.504 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ grep -i Replaying 2023-08-26T13:41:43.504 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: "state": "Replaying" 2023-08-26T13:41:43.511 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ current_state=started 2023-08-26T13:41:43.511 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ test started = started 2023-08-26T13:41:43.512 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ return 0 2023-08-26T13:41:43.512 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror test 2023-08-26T13:41:43.512 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local local_cluster=cluster1 2023-08-26T13:41:43.512 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster2 2023-08-26T13:41:43.513 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T13:41:43.513 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=test 2023-08-26T13:41:43.513 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ '[' snapshot = journal ']' 2023-08-26T13:41:43.513 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ '[' snapshot = snapshot ']' 2023-08-26T13:41:43.513 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror test 2023-08-26T13:41:43.514 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local local_cluster=cluster1 2023-08-26T13:41:43.514 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster2 2023-08-26T13:41:43.514 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T13:41:43.514 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=test 2023-08-26T13:41:43.514 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ mkfname cluster2-mirror-test.status 2023-08-26T13:41:43.515 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ echo cluster2-mirror-test.status 2023-08-26T13:41:43.515 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ sed -e 's|[/ ]|_|g' 2023-08-26T13:41:43.515 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local status_log=/tmp/tmp.TfDn6BcQ3N/cluster2-mirror-test.status 2023-08-26T13:41:43.515 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ mkfname cluster1-mirror-test.status 2023-08-26T13:41:43.515 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ echo cluster1-mirror-test.status 2023-08-26T13:41:43.515 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ sed -e 's|[/ ]|_|g' 2023-08-26T13:41:43.516 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local local_status_log=/tmp/tmp.TfDn6BcQ3N/cluster1-mirror-test.status 2023-08-26T13:41:43.516 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ mirror_image_snapshot cluster2 mirror test 2023-08-26T13:41:43.516 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster2 2023-08-26T13:41:43.516 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T13:41:43.516 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=test 2023-08-26T13:41:43.517 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/test 2023-08-26T13:41:45.018 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:Snapshot ID: 41 2023-08-26T13:41:45.037 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ get_newest_mirror_snapshot cluster2 mirror test /tmp/tmp.TfDn6BcQ3N/cluster2-mirror-test.status 2023-08-26T13:41:45.037 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster2 2023-08-26T13:41:45.037 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T13:41:45.038 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=test 2023-08-26T13:41:45.038 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local log=/tmp/tmp.TfDn6BcQ3N/cluster2-mirror-test.status 2023-08-26T13:41:45.038 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-08-26T13:41:45.038 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rbd --cluster cluster2 snap list --all mirror/test --format xml 2023-08-26T13:41:45.101 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ xmlstarlet sel -t -v //snapshot/id 2023-08-26T13:41:45.104 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local snapshot_id=41 2023-08-26T13:41:45.104 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ true 2023-08-26T13:41:45.104 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-08-26T13:41:45.104 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ sleep 0.2 2023-08-26T13:41:45.305 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.TfDn6BcQ3N/cluster1-mirror-test.status 2023-08-26T13:41:45.306 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster1 2023-08-26T13:41:45.306 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T13:41:45.306 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=test 2023-08-26T13:41:45.306 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local log=/tmp/tmp.TfDn6BcQ3N/cluster1-mirror-test.status 2023-08-26T13:41:45.307 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2023-08-26T13:41:45.307 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-08-26T13:41:45.354 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ true 2023-08-26T13:41:45.354 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-08-26T13:41:45.355 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:-:1.1: Document is empty 2023-08-26T13:41:45.355 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr: 2023-08-26T13:41:45.356 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:^ 2023-08-26T13:41:45.356 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local primary_snapshot_id= 2023-08-26T13:41:45.357 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ test 41 = '' 2023-08-26T13:41:45.357 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-08-26T13:41:45.357 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ sleep 0.4 2023-08-26T13:41:45.757 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.TfDn6BcQ3N/cluster1-mirror-test.status 2023-08-26T13:41:45.757 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster1 2023-08-26T13:41:45.757 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T13:41:45.758 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=test 2023-08-26T13:41:45.758 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local log=/tmp/tmp.TfDn6BcQ3N/cluster1-mirror-test.status 2023-08-26T13:41:45.758 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2023-08-26T13:41:45.758 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-08-26T13:41:45.807 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ true 2023-08-26T13:41:45.807 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-08-26T13:41:45.808 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:-:1.1: Document is empty 2023-08-26T13:41:45.808 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr: 2023-08-26T13:41:45.808 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:^ 2023-08-26T13:41:45.809 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local primary_snapshot_id= 2023-08-26T13:41:45.809 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ test 41 = '' 2023-08-26T13:41:45.809 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-08-26T13:41:45.810 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ sleep 0.8 2023-08-26T13:41:46.610 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.TfDn6BcQ3N/cluster1-mirror-test.status 2023-08-26T13:41:46.610 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster1 2023-08-26T13:41:46.610 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T13:41:46.611 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=test 2023-08-26T13:41:46.611 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local log=/tmp/tmp.TfDn6BcQ3N/cluster1-mirror-test.status 2023-08-26T13:41:46.611 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2023-08-26T13:41:46.611 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-08-26T13:41:46.719 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ true 2023-08-26T13:41:46.719 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-08-26T13:41:46.720 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:-:1.1: Document is empty 2023-08-26T13:41:46.720 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr: 2023-08-26T13:41:46.721 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:^ 2023-08-26T13:41:46.721 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local primary_snapshot_id= 2023-08-26T13:41:46.722 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ test 41 = '' 2023-08-26T13:41:46.722 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-08-26T13:41:46.722 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ sleep 1.6 2023-08-26T13:41:48.322 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.TfDn6BcQ3N/cluster1-mirror-test.status 2023-08-26T13:41:48.323 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster1 2023-08-26T13:41:48.323 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T13:41:48.323 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=test 2023-08-26T13:41:48.324 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local log=/tmp/tmp.TfDn6BcQ3N/cluster1-mirror-test.status 2023-08-26T13:41:48.324 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2023-08-26T13:41:48.324 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-08-26T13:41:48.402 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-08-26T13:41:48.404 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local primary_snapshot_id=41 2023-08-26T13:41:48.405 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ test 41 = 41 2023-08-26T13:41:48.405 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ return 0 2023-08-26T13:41:48.405 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ wait_for_snap_present cluster1 mirror test snap10 2023-08-26T13:41:48.405 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster1 2023-08-26T13:41:48.406 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T13:41:48.406 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=test 2023-08-26T13:41:48.406 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local snap_name=snap10 2023-08-26T13:41:48.406 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local s 2023-08-26T13:41:48.406 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2023-08-26T13:41:48.407 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ sleep 1 2023-08-26T13:41:49.405 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rbd --cluster cluster1 info mirror/test@snap10 2023-08-26T13:41:49.453 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:rbd image 'test': 2023-08-26T13:41:49.454 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: size 512 MiB in 128 objects 2023-08-26T13:41:49.454 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: order 22 (4 MiB objects) 2023-08-26T13:41:49.454 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: snapshot_count: 20 2023-08-26T13:41:49.454 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: id: 10ab632cdafc 2023-08-26T13:41:49.455 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: block_name_prefix: rbd_data.10ab632cdafc 2023-08-26T13:41:49.455 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: format: 2 2023-08-26T13:41:49.455 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: features: layering, exclusive-lock, non-primary 2023-08-26T13:41:49.455 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: op_features: 2023-08-26T13:41:49.455 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: flags: 2023-08-26T13:41:49.456 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: create_timestamp: Sat Aug 26 13:33:17 2023 2023-08-26T13:41:49.456 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: access_timestamp: Sat Aug 26 13:40:56 2023 2023-08-26T13:41:49.456 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: modify_timestamp: Sat Aug 26 13:33:17 2023 2023-08-26T13:41:49.456 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: protected: False 2023-08-26T13:41:49.456 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: mirroring state: enabled 2023-08-26T13:41:49.457 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: mirroring mode: snapshot 2023-08-26T13:41:49.457 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: mirroring global id: e2f2554d-659e-4467-b7f8-dddd7ccb0ee9 2023-08-26T13:41:49.457 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: mirroring primary: false 2023-08-26T13:41:49.458 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ return 0 2023-08-26T13:41:49.459 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ '[' -n snap9-clean ']' 2023-08-26T13:41:49.459 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ compare_image_snaps mirror test snap9-clean 2023-08-26T13:41:49.459 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T13:41:49.459 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=test 2023-08-26T13:41:49.459 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local snap_name=snap9-clean 2023-08-26T13:41:49.460 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local ret=0 2023-08-26T13:41:49.460 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local rmt_export=/tmp/tmp.TfDn6BcQ3N/cluster2-mirror-test.export 2023-08-26T13:41:49.460 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local loc_export=/tmp/tmp.TfDn6BcQ3N/cluster1-mirror-test.export 2023-08-26T13:41:49.460 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rm -f /tmp/tmp.TfDn6BcQ3N/cluster2-mirror-test.export /tmp/tmp.TfDn6BcQ3N/cluster1-mirror-test.export 2023-08-26T13:41:49.460 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rbd --cluster cluster2 -p mirror export test@snap9-clean /tmp/tmp.TfDn6BcQ3N/cluster2-mirror-test.export 2023-08-26T13:41:50.053 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2023-08-26T13:41:50.063 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rbd --cluster cluster1 -p mirror export test@snap9-clean /tmp/tmp.TfDn6BcQ3N/cluster1-mirror-test.export 2023-08-26T13:41:50.719 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2023-08-26T13:41:50.728 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ cmp /tmp/tmp.TfDn6BcQ3N/cluster2-mirror-test.export /tmp/tmp.TfDn6BcQ3N/cluster1-mirror-test.export 2023-08-26T13:41:50.940 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rm -f /tmp/tmp.TfDn6BcQ3N/cluster2-mirror-test.export /tmp/tmp.TfDn6BcQ3N/cluster1-mirror-test.export 2023-08-26T13:41:51.083 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ return 0 2023-08-26T13:41:51.084 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ compare_image_snaps mirror test snap10 2023-08-26T13:41:51.084 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T13:41:51.084 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=test 2023-08-26T13:41:51.084 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local snap_name=snap10 2023-08-26T13:41:51.084 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local ret=0 2023-08-26T13:41:51.085 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local rmt_export=/tmp/tmp.TfDn6BcQ3N/cluster2-mirror-test.export 2023-08-26T13:41:51.085 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local loc_export=/tmp/tmp.TfDn6BcQ3N/cluster1-mirror-test.export 2023-08-26T13:41:51.085 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rm -f /tmp/tmp.TfDn6BcQ3N/cluster2-mirror-test.export /tmp/tmp.TfDn6BcQ3N/cluster1-mirror-test.export 2023-08-26T13:41:51.085 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rbd --cluster cluster2 -p mirror export test@snap10 /tmp/tmp.TfDn6BcQ3N/cluster2-mirror-test.export 2023-08-26T13:41:51.670 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2023-08-26T13:41:51.682 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rbd --cluster cluster1 -p mirror export test@snap10 /tmp/tmp.TfDn6BcQ3N/cluster1-mirror-test.export 2023-08-26T13:41:52.374 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2023-08-26T13:41:52.384 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ cmp /tmp/tmp.TfDn6BcQ3N/cluster2-mirror-test.export /tmp/tmp.TfDn6BcQ3N/cluster1-mirror-test.export 2023-08-26T13:41:52.598 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rm -f /tmp/tmp.TfDn6BcQ3N/cluster2-mirror-test.export /tmp/tmp.TfDn6BcQ3N/cluster1-mirror-test.export 2023-08-26T13:41:52.743 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ return 0 2023-08-26T13:41:52.743 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ clean_snap_name=snap10-clean 2023-08-26T13:41:52.743 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ create_snap cluster2 mirror test snap10-clean 2023-08-26T13:41:52.743 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster2 2023-08-26T13:41:52.744 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T13:41:52.744 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=test 2023-08-26T13:41:52.744 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local snap_name=snap10-clean 2023-08-26T13:41:52.744 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rbd --cluster cluster2 -p mirror snap create test@snap10-clean --debug-rbd=20 --debug-journaler=20 2023-08-26T13:41:53.047 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ wait_for_image_replay_started cluster1 mirror test 2023-08-26T13:41:53.047 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster1 2023-08-26T13:41:53.048 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T13:41:53.048 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=test 2023-08-26T13:41:53.048 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ wait_for_image_replay_state cluster1 mirror test started 2023-08-26T13:41:53.048 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster1 2023-08-26T13:41:53.048 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T13:41:53.049 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=test 2023-08-26T13:41:53.049 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local state=started 2023-08-26T13:41:53.049 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local s 2023-08-26T13:41:53.049 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-08-26T13:41:53.049 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ sleep 1 2023-08-26T13:41:54.050 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ test_image_replay_state cluster1 mirror test started 2023-08-26T13:41:54.050 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster1 2023-08-26T13:41:54.050 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T13:41:54.051 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=test 2023-08-26T13:41:54.051 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local test_state=started 2023-08-26T13:41:54.051 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local status_result 2023-08-26T13:41:54.052 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local current_state=stopped 2023-08-26T13:41:54.052 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ admin_daemons cluster1 rbd mirror status mirror/test 2023-08-26T13:41:54.052 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local cluster_instance=cluster1 2023-08-26T13:41:54.052 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ shift 2023-08-26T13:41:54.052 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ grep -i state 2023-08-26T13:41:54.053 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local cluster=cluster1 2023-08-26T13:41:54.053 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local instance=cluster1 2023-08-26T13:41:54.053 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local loop_instance 2023-08-26T13:41:54.053 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-08-26T13:41:54.053 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ sleep 0 2023-08-26T13:41:54.054 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ '[' cluster1 '!=' cluster1 ']' 2023-08-26T13:41:54.056 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ seq 0 3 2023-08-26T13:41:54.056 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-08-26T13:41:54.056 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test 2023-08-26T13:41:54.056 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local cluster=cluster1:0 2023-08-26T13:41:54.056 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ shift 2023-08-26T13:41:54.057 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local instance 2023-08-26T13:41:54.057 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ set_cluster_instance cluster1:0 cluster instance 2023-08-26T13:41:54.058 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local val=cluster1:0 2023-08-26T13:41:54.058 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local cluster_var_name=cluster 2023-08-26T13:41:54.058 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local instance_var_name=instance 2023-08-26T13:41:54.059 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ cluster=cluster1 2023-08-26T13:41:54.059 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ instance=0 2023-08-26T13:41:54.059 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ '[' 0 = cluster1:0 ']' 2023-08-26T13:41:54.059 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ eval cluster=cluster1 2023-08-26T13:41:54.059 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ cluster=cluster1 2023-08-26T13:41:54.060 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ eval instance=0 2023-08-26T13:41:54.060 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ instance=0 2023-08-26T13:41:54.060 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ daemon_asok_file cluster1:0 cluster1 2023-08-26T13:41:54.060 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ local local_cluster=cluster1:0 2023-08-26T13:41:54.060 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ local cluster=cluster1 2023-08-26T13:41:54.061 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ local instance 2023-08-26T13:41:54.061 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2023-08-26T13:41:54.062 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ local val=cluster1:0 2023-08-26T13:41:54.062 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ local cluster_var_name=local_cluster 2023-08-26T13:41:54.063 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ local instance_var_name=instance 2023-08-26T13:41:54.063 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ cluster=cluster1 2023-08-26T13:41:54.063 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ instance=0 2023-08-26T13:41:54.063 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ '[' 0 = cluster1:0 ']' 2023-08-26T13:41:54.064 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ eval local_cluster=cluster1 2023-08-26T13:41:54.064 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++++ local_cluster=cluster1 2023-08-26T13:41:54.064 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ eval instance=0 2023-08-26T13:41:54.064 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++++ instance=0 2023-08-26T13:41:54.064 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2023-08-26T13:41:54.075 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-08-26T13:41:54.075 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-08-26T13:41:54.076 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-08-26T13:41:54.076 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2023-08-26T13:41:54.180 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:Invalid command: unused arguments: ['mirror/test'] 2023-08-26T13:41:54.181 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:rbd mirror status : get status for rbd mirror 2023-08-26T13:41:54.181 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:admin_socket: invalid command 2023-08-26T13:41:54.184 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-08-26T13:41:54.184 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/test 2023-08-26T13:41:54.184 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local cluster=cluster1:1 2023-08-26T13:41:54.184 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ shift 2023-08-26T13:41:54.184 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local instance 2023-08-26T13:41:54.185 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ set_cluster_instance cluster1:1 cluster instance 2023-08-26T13:41:54.185 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local val=cluster1:1 2023-08-26T13:41:54.185 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local cluster_var_name=cluster 2023-08-26T13:41:54.185 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local instance_var_name=instance 2023-08-26T13:41:54.185 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ cluster=cluster1 2023-08-26T13:41:54.186 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ instance=1 2023-08-26T13:41:54.186 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ '[' 1 = cluster1:1 ']' 2023-08-26T13:41:54.186 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ eval cluster=cluster1 2023-08-26T13:41:54.186 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ cluster=cluster1 2023-08-26T13:41:54.186 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ eval instance=1 2023-08-26T13:41:54.186 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ instance=1 2023-08-26T13:41:54.187 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ daemon_asok_file cluster1:1 cluster1 2023-08-26T13:41:54.187 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ local local_cluster=cluster1:1 2023-08-26T13:41:54.187 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ local cluster=cluster1 2023-08-26T13:41:54.187 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ local instance 2023-08-26T13:41:54.187 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2023-08-26T13:41:54.188 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ local val=cluster1:1 2023-08-26T13:41:54.188 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ local cluster_var_name=local_cluster 2023-08-26T13:41:54.188 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ local instance_var_name=instance 2023-08-26T13:41:54.188 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ cluster=cluster1 2023-08-26T13:41:54.188 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ instance=1 2023-08-26T13:41:54.188 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ '[' 1 = cluster1:1 ']' 2023-08-26T13:41:54.189 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ eval local_cluster=cluster1 2023-08-26T13:41:54.189 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++++ local_cluster=cluster1 2023-08-26T13:41:54.189 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ eval instance=1 2023-08-26T13:41:54.189 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++++ instance=1 2023-08-26T13:41:54.189 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2023-08-26T13:41:54.203 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-08-26T13:41:54.203 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-08-26T13:41:54.203 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-08-26T13:41:54.204 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/test 2023-08-26T13:41:54.297 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:Invalid command: unused arguments: ['mirror/test'] 2023-08-26T13:41:54.298 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:rbd mirror status : get status for rbd mirror 2023-08-26T13:41:54.298 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:admin_socket: invalid command 2023-08-26T13:41:54.300 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-08-26T13:41:54.301 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/test 2023-08-26T13:41:54.301 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local cluster=cluster1:2 2023-08-26T13:41:54.301 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ shift 2023-08-26T13:41:54.301 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local instance 2023-08-26T13:41:54.301 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ set_cluster_instance cluster1:2 cluster instance 2023-08-26T13:41:54.302 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local val=cluster1:2 2023-08-26T13:41:54.302 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local cluster_var_name=cluster 2023-08-26T13:41:54.302 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local instance_var_name=instance 2023-08-26T13:41:54.302 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ cluster=cluster1 2023-08-26T13:41:54.302 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ instance=2 2023-08-26T13:41:54.303 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ '[' 2 = cluster1:2 ']' 2023-08-26T13:41:54.303 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ eval cluster=cluster1 2023-08-26T13:41:54.303 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ cluster=cluster1 2023-08-26T13:41:54.303 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ eval instance=2 2023-08-26T13:41:54.303 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ instance=2 2023-08-26T13:41:54.303 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ daemon_asok_file cluster1:2 cluster1 2023-08-26T13:41:54.304 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ local local_cluster=cluster1:2 2023-08-26T13:41:54.304 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ local cluster=cluster1 2023-08-26T13:41:54.304 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ local instance 2023-08-26T13:41:54.304 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2023-08-26T13:41:54.304 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ local val=cluster1:2 2023-08-26T13:41:54.305 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ local cluster_var_name=local_cluster 2023-08-26T13:41:54.305 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ local instance_var_name=instance 2023-08-26T13:41:54.305 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ cluster=cluster1 2023-08-26T13:41:54.306 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ instance=2 2023-08-26T13:41:54.306 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ '[' 2 = cluster1:2 ']' 2023-08-26T13:41:54.306 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ eval local_cluster=cluster1 2023-08-26T13:41:54.306 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++++ local_cluster=cluster1 2023-08-26T13:41:54.307 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ eval instance=2 2023-08-26T13:41:54.308 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++++ instance=2 2023-08-26T13:41:54.308 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2023-08-26T13:41:54.319 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-08-26T13:41:54.319 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-08-26T13:41:54.319 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-08-26T13:41:54.320 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/test 2023-08-26T13:41:54.412 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:Invalid command: unused arguments: ['mirror/test'] 2023-08-26T13:41:54.413 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:rbd mirror status : get status for rbd mirror 2023-08-26T13:41:54.413 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:admin_socket: invalid command 2023-08-26T13:41:54.416 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-08-26T13:41:54.416 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ admin_daemon cluster1:3 rbd mirror status mirror/test 2023-08-26T13:41:54.416 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local cluster=cluster1:3 2023-08-26T13:41:54.417 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ shift 2023-08-26T13:41:54.417 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local instance 2023-08-26T13:41:54.417 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ set_cluster_instance cluster1:3 cluster instance 2023-08-26T13:41:54.417 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local val=cluster1:3 2023-08-26T13:41:54.417 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local cluster_var_name=cluster 2023-08-26T13:41:54.417 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local instance_var_name=instance 2023-08-26T13:41:54.418 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ cluster=cluster1 2023-08-26T13:41:54.418 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ instance=3 2023-08-26T13:41:54.418 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ '[' 3 = cluster1:3 ']' 2023-08-26T13:41:54.418 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ eval cluster=cluster1 2023-08-26T13:41:54.418 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ cluster=cluster1 2023-08-26T13:41:54.419 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ eval instance=3 2023-08-26T13:41:54.419 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ instance=3 2023-08-26T13:41:54.419 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ daemon_asok_file cluster1:3 cluster1 2023-08-26T13:41:54.419 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ local local_cluster=cluster1:3 2023-08-26T13:41:54.419 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ local cluster=cluster1 2023-08-26T13:41:54.419 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ local instance 2023-08-26T13:41:54.420 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ set_cluster_instance cluster1:3 local_cluster instance 2023-08-26T13:41:54.420 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ local val=cluster1:3 2023-08-26T13:41:54.420 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ local cluster_var_name=local_cluster 2023-08-26T13:41:54.420 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ local instance_var_name=instance 2023-08-26T13:41:54.420 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ cluster=cluster1 2023-08-26T13:41:54.421 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ instance=3 2023-08-26T13:41:54.421 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ '[' 3 = cluster1:3 ']' 2023-08-26T13:41:54.421 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ eval local_cluster=cluster1 2023-08-26T13:41:54.421 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++++ local_cluster=cluster1 2023-08-26T13:41:54.421 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ eval instance=3 2023-08-26T13:41:54.422 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++++ instance=3 2023-08-26T13:41:54.422 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2023-08-26T13:41:54.434 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2023-08-26T13:41:54.434 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2023-08-26T13:41:54.434 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2023-08-26T13:41:54.435 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror status mirror/test 2023-08-26T13:41:54.538 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ return 0 2023-08-26T13:41:54.539 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ status_result=' "state": "Replaying"' 2023-08-26T13:41:54.540 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ echo ' "state": "Replaying"' 2023-08-26T13:41:54.540 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ grep -i Replaying 2023-08-26T13:41:54.540 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: "state": "Replaying" 2023-08-26T13:41:54.544 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ current_state=started 2023-08-26T13:41:54.544 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ test started = started 2023-08-26T13:41:54.544 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ return 0 2023-08-26T13:41:54.545 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror test 2023-08-26T13:41:54.545 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local local_cluster=cluster1 2023-08-26T13:41:54.545 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster2 2023-08-26T13:41:54.545 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T13:41:54.545 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=test 2023-08-26T13:41:54.546 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ '[' snapshot = journal ']' 2023-08-26T13:41:54.546 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ '[' snapshot = snapshot ']' 2023-08-26T13:41:54.546 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror test 2023-08-26T13:41:54.546 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local local_cluster=cluster1 2023-08-26T13:41:54.547 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster2 2023-08-26T13:41:54.547 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T13:41:54.547 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=test 2023-08-26T13:41:54.547 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ mkfname cluster2-mirror-test.status 2023-08-26T13:41:54.547 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ echo cluster2-mirror-test.status 2023-08-26T13:41:54.548 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ sed -e 's|[/ ]|_|g' 2023-08-26T13:41:54.548 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local status_log=/tmp/tmp.TfDn6BcQ3N/cluster2-mirror-test.status 2023-08-26T13:41:54.548 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ mkfname cluster1-mirror-test.status 2023-08-26T13:41:54.548 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ echo cluster1-mirror-test.status 2023-08-26T13:41:54.549 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ sed -e 's|[/ ]|_|g' 2023-08-26T13:41:54.549 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local local_status_log=/tmp/tmp.TfDn6BcQ3N/cluster1-mirror-test.status 2023-08-26T13:41:54.549 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ mirror_image_snapshot cluster2 mirror test 2023-08-26T13:41:54.549 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster2 2023-08-26T13:41:54.550 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T13:41:54.550 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=test 2023-08-26T13:41:54.550 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/test 2023-08-26T13:41:56.060 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:Snapshot ID: 44 2023-08-26T13:41:56.079 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ get_newest_mirror_snapshot cluster2 mirror test /tmp/tmp.TfDn6BcQ3N/cluster2-mirror-test.status 2023-08-26T13:41:56.079 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster2 2023-08-26T13:41:56.079 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T13:41:56.080 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=test 2023-08-26T13:41:56.080 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local log=/tmp/tmp.TfDn6BcQ3N/cluster2-mirror-test.status 2023-08-26T13:41:56.082 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rbd --cluster cluster2 snap list --all mirror/test --format xml 2023-08-26T13:41:56.082 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-08-26T13:41:56.161 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ xmlstarlet sel -t -v //snapshot/id 2023-08-26T13:41:56.164 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local snapshot_id=44 2023-08-26T13:41:56.164 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ true 2023-08-26T13:41:56.165 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-08-26T13:41:56.165 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ sleep 0.2 2023-08-26T13:41:56.366 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.TfDn6BcQ3N/cluster1-mirror-test.status 2023-08-26T13:41:56.366 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster1 2023-08-26T13:41:56.366 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T13:41:56.367 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=test 2023-08-26T13:41:56.367 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local log=/tmp/tmp.TfDn6BcQ3N/cluster1-mirror-test.status 2023-08-26T13:41:56.367 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2023-08-26T13:41:56.368 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-08-26T13:41:56.415 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-08-26T13:41:56.418 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local primary_snapshot_id=44 2023-08-26T13:41:56.418 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ test 44 = 44 2023-08-26T13:41:56.418 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ return 0 2023-08-26T13:41:56.419 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ wait_for_snap_present cluster1 mirror test snap10-clean 2023-08-26T13:41:56.419 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster1 2023-08-26T13:41:56.419 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T13:41:56.419 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=test 2023-08-26T13:41:56.420 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local snap_name=snap10-clean 2023-08-26T13:41:56.420 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local s 2023-08-26T13:41:56.420 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2023-08-26T13:41:56.420 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ sleep 1 2023-08-26T13:41:57.418 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rbd --cluster cluster1 info mirror/test@snap10-clean 2023-08-26T13:41:57.469 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:rbd image 'test': 2023-08-26T13:41:57.470 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: size 512 MiB in 128 objects 2023-08-26T13:41:57.470 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: order 22 (4 MiB objects) 2023-08-26T13:41:57.470 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: snapshot_count: 21 2023-08-26T13:41:57.470 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: id: 10ab632cdafc 2023-08-26T13:41:57.471 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: block_name_prefix: rbd_data.10ab632cdafc 2023-08-26T13:41:57.471 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: format: 2 2023-08-26T13:41:57.471 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: features: layering, exclusive-lock, non-primary 2023-08-26T13:41:57.471 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: op_features: 2023-08-26T13:41:57.471 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: flags: 2023-08-26T13:41:57.472 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: create_timestamp: Sat Aug 26 13:33:17 2023 2023-08-26T13:41:57.472 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: access_timestamp: Sat Aug 26 13:40:56 2023 2023-08-26T13:41:57.472 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: modify_timestamp: Sat Aug 26 13:33:17 2023 2023-08-26T13:41:57.472 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: protected: False 2023-08-26T13:41:57.472 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: mirroring state: enabled 2023-08-26T13:41:57.473 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: mirroring mode: snapshot 2023-08-26T13:41:57.473 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: mirroring global id: e2f2554d-659e-4467-b7f8-dddd7ccb0ee9 2023-08-26T13:41:57.473 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: mirroring primary: false 2023-08-26T13:41:57.474 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ return 0 2023-08-26T13:41:57.474 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ seq 1 10 2023-08-26T13:41:57.475 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ for i in `seq 1 10` 2023-08-26T13:41:57.475 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ snap_name=snap1 2023-08-26T13:41:57.475 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ compare_image_snaps mirror test snap1 2023-08-26T13:41:57.476 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T13:41:57.476 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=test 2023-08-26T13:41:57.476 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local snap_name=snap1 2023-08-26T13:41:57.476 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local ret=0 2023-08-26T13:41:57.476 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local rmt_export=/tmp/tmp.TfDn6BcQ3N/cluster2-mirror-test.export 2023-08-26T13:41:57.477 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local loc_export=/tmp/tmp.TfDn6BcQ3N/cluster1-mirror-test.export 2023-08-26T13:41:57.477 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rm -f /tmp/tmp.TfDn6BcQ3N/cluster2-mirror-test.export /tmp/tmp.TfDn6BcQ3N/cluster1-mirror-test.export 2023-08-26T13:41:57.477 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rbd --cluster cluster2 -p mirror export test@snap1 /tmp/tmp.TfDn6BcQ3N/cluster2-mirror-test.export 2023-08-26T13:41:58.089 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2023-08-26T13:41:58.098 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rbd --cluster cluster1 -p mirror export test@snap1 /tmp/tmp.TfDn6BcQ3N/cluster1-mirror-test.export 2023-08-26T13:41:58.838 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2023-08-26T13:41:58.846 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ cmp /tmp/tmp.TfDn6BcQ3N/cluster2-mirror-test.export /tmp/tmp.TfDn6BcQ3N/cluster1-mirror-test.export 2023-08-26T13:41:59.056 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rm -f /tmp/tmp.TfDn6BcQ3N/cluster2-mirror-test.export /tmp/tmp.TfDn6BcQ3N/cluster1-mirror-test.export 2023-08-26T13:41:59.199 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ return 0 2023-08-26T13:41:59.199 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ snap_name=snap1-clean 2023-08-26T13:41:59.200 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ compare_image_snaps mirror test snap1-clean 2023-08-26T13:41:59.200 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T13:41:59.200 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=test 2023-08-26T13:41:59.200 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local snap_name=snap1-clean 2023-08-26T13:41:59.200 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local ret=0 2023-08-26T13:41:59.201 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local rmt_export=/tmp/tmp.TfDn6BcQ3N/cluster2-mirror-test.export 2023-08-26T13:41:59.201 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local loc_export=/tmp/tmp.TfDn6BcQ3N/cluster1-mirror-test.export 2023-08-26T13:41:59.201 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rm -f /tmp/tmp.TfDn6BcQ3N/cluster2-mirror-test.export /tmp/tmp.TfDn6BcQ3N/cluster1-mirror-test.export 2023-08-26T13:41:59.201 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rbd --cluster cluster2 -p mirror export test@snap1-clean /tmp/tmp.TfDn6BcQ3N/cluster2-mirror-test.export 2023-08-26T13:41:59.789 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2023-08-26T13:41:59.801 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rbd --cluster cluster1 -p mirror export test@snap1-clean /tmp/tmp.TfDn6BcQ3N/cluster1-mirror-test.export 2023-08-26T13:42:00.540 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2023-08-26T13:42:00.547 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ cmp /tmp/tmp.TfDn6BcQ3N/cluster2-mirror-test.export /tmp/tmp.TfDn6BcQ3N/cluster1-mirror-test.export 2023-08-26T13:42:00.763 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rm -f /tmp/tmp.TfDn6BcQ3N/cluster2-mirror-test.export /tmp/tmp.TfDn6BcQ3N/cluster1-mirror-test.export 2023-08-26T13:42:00.906 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ return 0 2023-08-26T13:42:00.907 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ for i in `seq 1 10` 2023-08-26T13:42:00.907 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ snap_name=snap2 2023-08-26T13:42:00.907 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ compare_image_snaps mirror test snap2 2023-08-26T13:42:00.908 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T13:42:00.908 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=test 2023-08-26T13:42:00.908 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local snap_name=snap2 2023-08-26T13:42:00.908 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local ret=0 2023-08-26T13:42:00.908 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local rmt_export=/tmp/tmp.TfDn6BcQ3N/cluster2-mirror-test.export 2023-08-26T13:42:00.909 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local loc_export=/tmp/tmp.TfDn6BcQ3N/cluster1-mirror-test.export 2023-08-26T13:42:00.909 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rm -f /tmp/tmp.TfDn6BcQ3N/cluster2-mirror-test.export /tmp/tmp.TfDn6BcQ3N/cluster1-mirror-test.export 2023-08-26T13:42:00.909 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rbd --cluster cluster2 -p mirror export test@snap2 /tmp/tmp.TfDn6BcQ3N/cluster2-mirror-test.export 2023-08-26T13:42:01.503 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2023-08-26T13:42:01.513 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rbd --cluster cluster1 -p mirror export test@snap2 /tmp/tmp.TfDn6BcQ3N/cluster1-mirror-test.export 2023-08-26T13:42:02.256 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2023-08-26T13:42:02.264 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ cmp /tmp/tmp.TfDn6BcQ3N/cluster2-mirror-test.export /tmp/tmp.TfDn6BcQ3N/cluster1-mirror-test.export 2023-08-26T13:42:02.479 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rm -f /tmp/tmp.TfDn6BcQ3N/cluster2-mirror-test.export /tmp/tmp.TfDn6BcQ3N/cluster1-mirror-test.export 2023-08-26T13:42:02.625 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ return 0 2023-08-26T13:42:02.625 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ snap_name=snap2-clean 2023-08-26T13:42:02.626 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ compare_image_snaps mirror test snap2-clean 2023-08-26T13:42:02.626 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T13:42:02.626 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=test 2023-08-26T13:42:02.626 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local snap_name=snap2-clean 2023-08-26T13:42:02.627 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local ret=0 2023-08-26T13:42:02.627 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local rmt_export=/tmp/tmp.TfDn6BcQ3N/cluster2-mirror-test.export 2023-08-26T13:42:02.627 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local loc_export=/tmp/tmp.TfDn6BcQ3N/cluster1-mirror-test.export 2023-08-26T13:42:02.627 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rm -f /tmp/tmp.TfDn6BcQ3N/cluster2-mirror-test.export /tmp/tmp.TfDn6BcQ3N/cluster1-mirror-test.export 2023-08-26T13:42:02.628 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rbd --cluster cluster2 -p mirror export test@snap2-clean /tmp/tmp.TfDn6BcQ3N/cluster2-mirror-test.export 2023-08-26T13:42:03.195 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2023-08-26T13:42:03.207 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rbd --cluster cluster1 -p mirror export test@snap2-clean /tmp/tmp.TfDn6BcQ3N/cluster1-mirror-test.export 2023-08-26T13:42:03.976 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2023-08-26T13:42:03.985 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ cmp /tmp/tmp.TfDn6BcQ3N/cluster2-mirror-test.export /tmp/tmp.TfDn6BcQ3N/cluster1-mirror-test.export 2023-08-26T13:42:04.193 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rm -f /tmp/tmp.TfDn6BcQ3N/cluster2-mirror-test.export /tmp/tmp.TfDn6BcQ3N/cluster1-mirror-test.export 2023-08-26T13:42:04.335 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ return 0 2023-08-26T13:42:04.336 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ for i in `seq 1 10` 2023-08-26T13:42:04.336 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ snap_name=snap3 2023-08-26T13:42:04.336 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ compare_image_snaps mirror test snap3 2023-08-26T13:42:04.336 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T13:42:04.336 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=test 2023-08-26T13:42:04.337 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local snap_name=snap3 2023-08-26T13:42:04.337 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local ret=0 2023-08-26T13:42:04.337 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local rmt_export=/tmp/tmp.TfDn6BcQ3N/cluster2-mirror-test.export 2023-08-26T13:42:04.337 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local loc_export=/tmp/tmp.TfDn6BcQ3N/cluster1-mirror-test.export 2023-08-26T13:42:04.337 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rm -f /tmp/tmp.TfDn6BcQ3N/cluster2-mirror-test.export /tmp/tmp.TfDn6BcQ3N/cluster1-mirror-test.export 2023-08-26T13:42:04.338 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rbd --cluster cluster2 -p mirror export test@snap3 /tmp/tmp.TfDn6BcQ3N/cluster2-mirror-test.export 2023-08-26T13:42:04.933 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2023-08-26T13:42:04.942 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rbd --cluster cluster1 -p mirror export test@snap3 /tmp/tmp.TfDn6BcQ3N/cluster1-mirror-test.export 2023-08-26T13:42:05.667 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2023-08-26T13:42:05.676 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ cmp /tmp/tmp.TfDn6BcQ3N/cluster2-mirror-test.export /tmp/tmp.TfDn6BcQ3N/cluster1-mirror-test.export 2023-08-26T13:42:05.891 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rm -f /tmp/tmp.TfDn6BcQ3N/cluster2-mirror-test.export /tmp/tmp.TfDn6BcQ3N/cluster1-mirror-test.export 2023-08-26T13:42:06.035 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ return 0 2023-08-26T13:42:06.036 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ snap_name=snap3-clean 2023-08-26T13:42:06.036 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ compare_image_snaps mirror test snap3-clean 2023-08-26T13:42:06.036 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T13:42:06.036 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=test 2023-08-26T13:42:06.036 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local snap_name=snap3-clean 2023-08-26T13:42:06.037 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local ret=0 2023-08-26T13:42:06.037 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local rmt_export=/tmp/tmp.TfDn6BcQ3N/cluster2-mirror-test.export 2023-08-26T13:42:06.037 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local loc_export=/tmp/tmp.TfDn6BcQ3N/cluster1-mirror-test.export 2023-08-26T13:42:06.037 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rm -f /tmp/tmp.TfDn6BcQ3N/cluster2-mirror-test.export /tmp/tmp.TfDn6BcQ3N/cluster1-mirror-test.export 2023-08-26T13:42:06.037 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rbd --cluster cluster2 -p mirror export test@snap3-clean /tmp/tmp.TfDn6BcQ3N/cluster2-mirror-test.export 2023-08-26T13:42:06.592 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2023-08-26T13:42:06.602 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rbd --cluster cluster1 -p mirror export test@snap3-clean /tmp/tmp.TfDn6BcQ3N/cluster1-mirror-test.export 2023-08-26T13:42:07.374 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2023-08-26T13:42:07.384 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ cmp /tmp/tmp.TfDn6BcQ3N/cluster2-mirror-test.export /tmp/tmp.TfDn6BcQ3N/cluster1-mirror-test.export 2023-08-26T13:42:07.596 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rm -f /tmp/tmp.TfDn6BcQ3N/cluster2-mirror-test.export /tmp/tmp.TfDn6BcQ3N/cluster1-mirror-test.export 2023-08-26T13:42:07.741 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ return 0 2023-08-26T13:42:07.741 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ for i in `seq 1 10` 2023-08-26T13:42:07.741 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ snap_name=snap4 2023-08-26T13:42:07.741 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ compare_image_snaps mirror test snap4 2023-08-26T13:42:07.741 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T13:42:07.742 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=test 2023-08-26T13:42:07.742 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local snap_name=snap4 2023-08-26T13:42:07.742 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local ret=0 2023-08-26T13:42:07.742 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local rmt_export=/tmp/tmp.TfDn6BcQ3N/cluster2-mirror-test.export 2023-08-26T13:42:07.742 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local loc_export=/tmp/tmp.TfDn6BcQ3N/cluster1-mirror-test.export 2023-08-26T13:42:07.743 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rm -f /tmp/tmp.TfDn6BcQ3N/cluster2-mirror-test.export /tmp/tmp.TfDn6BcQ3N/cluster1-mirror-test.export 2023-08-26T13:42:07.743 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rbd --cluster cluster2 -p mirror export test@snap4 /tmp/tmp.TfDn6BcQ3N/cluster2-mirror-test.export 2023-08-26T13:42:08.335 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2023-08-26T13:42:08.345 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rbd --cluster cluster1 -p mirror export test@snap4 /tmp/tmp.TfDn6BcQ3N/cluster1-mirror-test.export 2023-08-26T13:42:09.105 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2023-08-26T13:42:09.114 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ cmp /tmp/tmp.TfDn6BcQ3N/cluster2-mirror-test.export /tmp/tmp.TfDn6BcQ3N/cluster1-mirror-test.export 2023-08-26T13:42:09.327 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rm -f /tmp/tmp.TfDn6BcQ3N/cluster2-mirror-test.export /tmp/tmp.TfDn6BcQ3N/cluster1-mirror-test.export 2023-08-26T13:42:09.469 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ return 0 2023-08-26T13:42:09.470 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ snap_name=snap4-clean 2023-08-26T13:42:09.470 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ compare_image_snaps mirror test snap4-clean 2023-08-26T13:42:09.470 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T13:42:09.470 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=test 2023-08-26T13:42:09.471 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local snap_name=snap4-clean 2023-08-26T13:42:09.471 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local ret=0 2023-08-26T13:42:09.471 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local rmt_export=/tmp/tmp.TfDn6BcQ3N/cluster2-mirror-test.export 2023-08-26T13:42:09.471 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local loc_export=/tmp/tmp.TfDn6BcQ3N/cluster1-mirror-test.export 2023-08-26T13:42:09.471 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rm -f /tmp/tmp.TfDn6BcQ3N/cluster2-mirror-test.export /tmp/tmp.TfDn6BcQ3N/cluster1-mirror-test.export 2023-08-26T13:42:09.472 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rbd --cluster cluster2 -p mirror export test@snap4-clean /tmp/tmp.TfDn6BcQ3N/cluster2-mirror-test.export 2023-08-26T13:42:10.059 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2023-08-26T13:42:10.068 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rbd --cluster cluster1 -p mirror export test@snap4-clean /tmp/tmp.TfDn6BcQ3N/cluster1-mirror-test.export 2023-08-26T13:42:10.822 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2023-08-26T13:42:10.829 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ cmp /tmp/tmp.TfDn6BcQ3N/cluster2-mirror-test.export /tmp/tmp.TfDn6BcQ3N/cluster1-mirror-test.export 2023-08-26T13:42:11.043 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rm -f /tmp/tmp.TfDn6BcQ3N/cluster2-mirror-test.export /tmp/tmp.TfDn6BcQ3N/cluster1-mirror-test.export 2023-08-26T13:42:11.186 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ return 0 2023-08-26T13:42:11.186 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ for i in `seq 1 10` 2023-08-26T13:42:11.187 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ snap_name=snap5 2023-08-26T13:42:11.187 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ compare_image_snaps mirror test snap5 2023-08-26T13:42:11.187 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T13:42:11.187 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=test 2023-08-26T13:42:11.187 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local snap_name=snap5 2023-08-26T13:42:11.188 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local ret=0 2023-08-26T13:42:11.188 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local rmt_export=/tmp/tmp.TfDn6BcQ3N/cluster2-mirror-test.export 2023-08-26T13:42:11.188 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local loc_export=/tmp/tmp.TfDn6BcQ3N/cluster1-mirror-test.export 2023-08-26T13:42:11.188 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rm -f /tmp/tmp.TfDn6BcQ3N/cluster2-mirror-test.export /tmp/tmp.TfDn6BcQ3N/cluster1-mirror-test.export 2023-08-26T13:42:11.188 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rbd --cluster cluster2 -p mirror export test@snap5 /tmp/tmp.TfDn6BcQ3N/cluster2-mirror-test.export 2023-08-26T13:42:11.789 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2023-08-26T13:42:11.799 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rbd --cluster cluster1 -p mirror export test@snap5 /tmp/tmp.TfDn6BcQ3N/cluster1-mirror-test.export 2023-08-26T13:42:12.473 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2023-08-26T13:42:12.481 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ cmp /tmp/tmp.TfDn6BcQ3N/cluster2-mirror-test.export /tmp/tmp.TfDn6BcQ3N/cluster1-mirror-test.export 2023-08-26T13:42:12.695 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rm -f /tmp/tmp.TfDn6BcQ3N/cluster2-mirror-test.export /tmp/tmp.TfDn6BcQ3N/cluster1-mirror-test.export 2023-08-26T13:42:12.840 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ return 0 2023-08-26T13:42:12.840 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ snap_name=snap5-clean 2023-08-26T13:42:12.840 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ compare_image_snaps mirror test snap5-clean 2023-08-26T13:42:12.840 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T13:42:12.840 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=test 2023-08-26T13:42:12.841 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local snap_name=snap5-clean 2023-08-26T13:42:12.841 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local ret=0 2023-08-26T13:42:12.841 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local rmt_export=/tmp/tmp.TfDn6BcQ3N/cluster2-mirror-test.export 2023-08-26T13:42:12.841 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local loc_export=/tmp/tmp.TfDn6BcQ3N/cluster1-mirror-test.export 2023-08-26T13:42:12.841 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rm -f /tmp/tmp.TfDn6BcQ3N/cluster2-mirror-test.export /tmp/tmp.TfDn6BcQ3N/cluster1-mirror-test.export 2023-08-26T13:42:12.842 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rbd --cluster cluster2 -p mirror export test@snap5-clean /tmp/tmp.TfDn6BcQ3N/cluster2-mirror-test.export 2023-08-26T13:42:13.403 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2023-08-26T13:42:13.413 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rbd --cluster cluster1 -p mirror export test@snap5-clean /tmp/tmp.TfDn6BcQ3N/cluster1-mirror-test.export 2023-08-26T13:42:14.200 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2023-08-26T13:42:14.209 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ cmp /tmp/tmp.TfDn6BcQ3N/cluster2-mirror-test.export /tmp/tmp.TfDn6BcQ3N/cluster1-mirror-test.export 2023-08-26T13:42:14.420 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rm -f /tmp/tmp.TfDn6BcQ3N/cluster2-mirror-test.export /tmp/tmp.TfDn6BcQ3N/cluster1-mirror-test.export 2023-08-26T13:42:14.565 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ return 0 2023-08-26T13:42:14.565 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ for i in `seq 1 10` 2023-08-26T13:42:14.565 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ snap_name=snap6 2023-08-26T13:42:14.565 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ compare_image_snaps mirror test snap6 2023-08-26T13:42:14.566 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T13:42:14.566 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=test 2023-08-26T13:42:14.566 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local snap_name=snap6 2023-08-26T13:42:14.566 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local ret=0 2023-08-26T13:42:14.566 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local rmt_export=/tmp/tmp.TfDn6BcQ3N/cluster2-mirror-test.export 2023-08-26T13:42:14.567 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local loc_export=/tmp/tmp.TfDn6BcQ3N/cluster1-mirror-test.export 2023-08-26T13:42:14.567 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rm -f /tmp/tmp.TfDn6BcQ3N/cluster2-mirror-test.export /tmp/tmp.TfDn6BcQ3N/cluster1-mirror-test.export 2023-08-26T13:42:14.567 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rbd --cluster cluster2 -p mirror export test@snap6 /tmp/tmp.TfDn6BcQ3N/cluster2-mirror-test.export 2023-08-26T13:42:15.153 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2023-08-26T13:42:15.165 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rbd --cluster cluster1 -p mirror export test@snap6 /tmp/tmp.TfDn6BcQ3N/cluster1-mirror-test.export 2023-08-26T13:42:15.836 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2023-08-26T13:42:15.846 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ cmp /tmp/tmp.TfDn6BcQ3N/cluster2-mirror-test.export /tmp/tmp.TfDn6BcQ3N/cluster1-mirror-test.export 2023-08-26T13:42:16.060 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rm -f /tmp/tmp.TfDn6BcQ3N/cluster2-mirror-test.export /tmp/tmp.TfDn6BcQ3N/cluster1-mirror-test.export 2023-08-26T13:42:16.205 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ return 0 2023-08-26T13:42:16.205 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ snap_name=snap6-clean 2023-08-26T13:42:16.206 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ compare_image_snaps mirror test snap6-clean 2023-08-26T13:42:16.206 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T13:42:16.206 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=test 2023-08-26T13:42:16.206 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local snap_name=snap6-clean 2023-08-26T13:42:16.206 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local ret=0 2023-08-26T13:42:16.207 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local rmt_export=/tmp/tmp.TfDn6BcQ3N/cluster2-mirror-test.export 2023-08-26T13:42:16.207 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local loc_export=/tmp/tmp.TfDn6BcQ3N/cluster1-mirror-test.export 2023-08-26T13:42:16.207 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rm -f /tmp/tmp.TfDn6BcQ3N/cluster2-mirror-test.export /tmp/tmp.TfDn6BcQ3N/cluster1-mirror-test.export 2023-08-26T13:42:16.207 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rbd --cluster cluster2 -p mirror export test@snap6-clean /tmp/tmp.TfDn6BcQ3N/cluster2-mirror-test.export 2023-08-26T13:42:16.784 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2023-08-26T13:42:16.796 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rbd --cluster cluster1 -p mirror export test@snap6-clean /tmp/tmp.TfDn6BcQ3N/cluster1-mirror-test.export 2023-08-26T13:42:17.573 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2023-08-26T13:42:17.582 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ cmp /tmp/tmp.TfDn6BcQ3N/cluster2-mirror-test.export /tmp/tmp.TfDn6BcQ3N/cluster1-mirror-test.export 2023-08-26T13:42:17.799 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rm -f /tmp/tmp.TfDn6BcQ3N/cluster2-mirror-test.export /tmp/tmp.TfDn6BcQ3N/cluster1-mirror-test.export 2023-08-26T13:42:17.943 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ return 0 2023-08-26T13:42:17.943 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ for i in `seq 1 10` 2023-08-26T13:42:17.944 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ snap_name=snap7 2023-08-26T13:42:17.944 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ compare_image_snaps mirror test snap7 2023-08-26T13:42:17.944 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T13:42:17.944 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=test 2023-08-26T13:42:17.945 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local snap_name=snap7 2023-08-26T13:42:17.945 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local ret=0 2023-08-26T13:42:17.945 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local rmt_export=/tmp/tmp.TfDn6BcQ3N/cluster2-mirror-test.export 2023-08-26T13:42:17.945 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local loc_export=/tmp/tmp.TfDn6BcQ3N/cluster1-mirror-test.export 2023-08-26T13:42:17.945 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rm -f /tmp/tmp.TfDn6BcQ3N/cluster2-mirror-test.export /tmp/tmp.TfDn6BcQ3N/cluster1-mirror-test.export 2023-08-26T13:42:17.945 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rbd --cluster cluster2 -p mirror export test@snap7 /tmp/tmp.TfDn6BcQ3N/cluster2-mirror-test.export 2023-08-26T13:42:18.518 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2023-08-26T13:42:18.526 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rbd --cluster cluster1 -p mirror export test@snap7 /tmp/tmp.TfDn6BcQ3N/cluster1-mirror-test.export 2023-08-26T13:42:19.234 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2023-08-26T13:42:19.243 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ cmp /tmp/tmp.TfDn6BcQ3N/cluster2-mirror-test.export /tmp/tmp.TfDn6BcQ3N/cluster1-mirror-test.export 2023-08-26T13:42:19.454 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rm -f /tmp/tmp.TfDn6BcQ3N/cluster2-mirror-test.export /tmp/tmp.TfDn6BcQ3N/cluster1-mirror-test.export 2023-08-26T13:42:19.600 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ return 0 2023-08-26T13:42:19.601 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ snap_name=snap7-clean 2023-08-26T13:42:19.601 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ compare_image_snaps mirror test snap7-clean 2023-08-26T13:42:19.601 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T13:42:19.601 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=test 2023-08-26T13:42:19.602 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local snap_name=snap7-clean 2023-08-26T13:42:19.602 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local ret=0 2023-08-26T13:42:19.602 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local rmt_export=/tmp/tmp.TfDn6BcQ3N/cluster2-mirror-test.export 2023-08-26T13:42:19.602 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local loc_export=/tmp/tmp.TfDn6BcQ3N/cluster1-mirror-test.export 2023-08-26T13:42:19.602 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rm -f /tmp/tmp.TfDn6BcQ3N/cluster2-mirror-test.export /tmp/tmp.TfDn6BcQ3N/cluster1-mirror-test.export 2023-08-26T13:42:19.603 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rbd --cluster cluster2 -p mirror export test@snap7-clean /tmp/tmp.TfDn6BcQ3N/cluster2-mirror-test.export 2023-08-26T13:42:20.215 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2023-08-26T13:42:20.224 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rbd --cluster cluster1 -p mirror export test@snap7-clean /tmp/tmp.TfDn6BcQ3N/cluster1-mirror-test.export 2023-08-26T13:42:21.182 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2023-08-26T13:42:21.191 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ cmp /tmp/tmp.TfDn6BcQ3N/cluster2-mirror-test.export /tmp/tmp.TfDn6BcQ3N/cluster1-mirror-test.export 2023-08-26T13:42:21.405 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rm -f /tmp/tmp.TfDn6BcQ3N/cluster2-mirror-test.export /tmp/tmp.TfDn6BcQ3N/cluster1-mirror-test.export 2023-08-26T13:42:21.550 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ return 0 2023-08-26T13:42:21.551 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ for i in `seq 1 10` 2023-08-26T13:42:21.551 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ snap_name=snap8 2023-08-26T13:42:21.551 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ compare_image_snaps mirror test snap8 2023-08-26T13:42:21.551 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T13:42:21.551 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=test 2023-08-26T13:42:21.552 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local snap_name=snap8 2023-08-26T13:42:21.552 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local ret=0 2023-08-26T13:42:21.552 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local rmt_export=/tmp/tmp.TfDn6BcQ3N/cluster2-mirror-test.export 2023-08-26T13:42:21.552 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local loc_export=/tmp/tmp.TfDn6BcQ3N/cluster1-mirror-test.export 2023-08-26T13:42:21.552 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rm -f /tmp/tmp.TfDn6BcQ3N/cluster2-mirror-test.export /tmp/tmp.TfDn6BcQ3N/cluster1-mirror-test.export 2023-08-26T13:42:21.553 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rbd --cluster cluster2 -p mirror export test@snap8 /tmp/tmp.TfDn6BcQ3N/cluster2-mirror-test.export 2023-08-26T13:42:22.113 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2023-08-26T13:42:22.123 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rbd --cluster cluster1 -p mirror export test@snap8 /tmp/tmp.TfDn6BcQ3N/cluster1-mirror-test.export 2023-08-26T13:42:22.589 INFO:tasks.rbd_mirror_thrash:kill cluster1.client.mirror.0 2023-08-26T13:42:22.590 INFO:tasks.rbd_mirror.cluster1.client.mirror.0:Sent signal 15 2023-08-26T13:42:22.590 INFO:tasks.rbd_mirror_thrash:waiting for 8 secs before reviving daemons 2023-08-26T13:42:22.591 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi124.stderr:2023-08-26T13:42:22.589+0000 7f5ae4c94700 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.0 (PID: 110421) UID: 1000 2023-08-26T13:42:22.931 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2023-08-26T13:42:22.940 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ cmp /tmp/tmp.TfDn6BcQ3N/cluster2-mirror-test.export /tmp/tmp.TfDn6BcQ3N/cluster1-mirror-test.export 2023-08-26T13:42:23.154 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rm -f /tmp/tmp.TfDn6BcQ3N/cluster2-mirror-test.export /tmp/tmp.TfDn6BcQ3N/cluster1-mirror-test.export 2023-08-26T13:42:23.298 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ return 0 2023-08-26T13:42:23.299 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ snap_name=snap8-clean 2023-08-26T13:42:23.299 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ compare_image_snaps mirror test snap8-clean 2023-08-26T13:42:23.299 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T13:42:23.300 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=test 2023-08-26T13:42:23.300 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local snap_name=snap8-clean 2023-08-26T13:42:23.300 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local ret=0 2023-08-26T13:42:23.300 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local rmt_export=/tmp/tmp.TfDn6BcQ3N/cluster2-mirror-test.export 2023-08-26T13:42:23.300 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local loc_export=/tmp/tmp.TfDn6BcQ3N/cluster1-mirror-test.export 2023-08-26T13:42:23.300 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rm -f /tmp/tmp.TfDn6BcQ3N/cluster2-mirror-test.export /tmp/tmp.TfDn6BcQ3N/cluster1-mirror-test.export 2023-08-26T13:42:23.301 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rbd --cluster cluster2 -p mirror export test@snap8-clean /tmp/tmp.TfDn6BcQ3N/cluster2-mirror-test.export 2023-08-26T13:42:23.897 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2023-08-26T13:42:23.909 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rbd --cluster cluster1 -p mirror export test@snap8-clean /tmp/tmp.TfDn6BcQ3N/cluster1-mirror-test.export 2023-08-26T13:42:24.572 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2023-08-26T13:42:24.580 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ cmp /tmp/tmp.TfDn6BcQ3N/cluster2-mirror-test.export /tmp/tmp.TfDn6BcQ3N/cluster1-mirror-test.export 2023-08-26T13:42:24.799 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rm -f /tmp/tmp.TfDn6BcQ3N/cluster2-mirror-test.export /tmp/tmp.TfDn6BcQ3N/cluster1-mirror-test.export 2023-08-26T13:42:24.944 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ return 0 2023-08-26T13:42:24.944 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ for i in `seq 1 10` 2023-08-26T13:42:24.944 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ snap_name=snap9 2023-08-26T13:42:24.944 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ compare_image_snaps mirror test snap9 2023-08-26T13:42:24.945 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T13:42:24.945 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=test 2023-08-26T13:42:24.945 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local snap_name=snap9 2023-08-26T13:42:24.945 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local ret=0 2023-08-26T13:42:24.945 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local rmt_export=/tmp/tmp.TfDn6BcQ3N/cluster2-mirror-test.export 2023-08-26T13:42:24.946 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local loc_export=/tmp/tmp.TfDn6BcQ3N/cluster1-mirror-test.export 2023-08-26T13:42:24.946 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rm -f /tmp/tmp.TfDn6BcQ3N/cluster2-mirror-test.export /tmp/tmp.TfDn6BcQ3N/cluster1-mirror-test.export 2023-08-26T13:42:24.946 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rbd --cluster cluster2 -p mirror export test@snap9 /tmp/tmp.TfDn6BcQ3N/cluster2-mirror-test.export 2023-08-26T13:42:25.529 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2023-08-26T13:42:25.539 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rbd --cluster cluster1 -p mirror export test@snap9 /tmp/tmp.TfDn6BcQ3N/cluster1-mirror-test.export 2023-08-26T13:42:26.288 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2023-08-26T13:42:26.297 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ cmp /tmp/tmp.TfDn6BcQ3N/cluster2-mirror-test.export /tmp/tmp.TfDn6BcQ3N/cluster1-mirror-test.export 2023-08-26T13:42:26.511 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rm -f /tmp/tmp.TfDn6BcQ3N/cluster2-mirror-test.export /tmp/tmp.TfDn6BcQ3N/cluster1-mirror-test.export 2023-08-26T13:42:26.657 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ return 0 2023-08-26T13:42:26.658 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ snap_name=snap9-clean 2023-08-26T13:42:26.658 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ compare_image_snaps mirror test snap9-clean 2023-08-26T13:42:26.658 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T13:42:26.658 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=test 2023-08-26T13:42:26.658 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local snap_name=snap9-clean 2023-08-26T13:42:26.659 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local ret=0 2023-08-26T13:42:26.659 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local rmt_export=/tmp/tmp.TfDn6BcQ3N/cluster2-mirror-test.export 2023-08-26T13:42:26.659 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local loc_export=/tmp/tmp.TfDn6BcQ3N/cluster1-mirror-test.export 2023-08-26T13:42:26.659 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rm -f /tmp/tmp.TfDn6BcQ3N/cluster2-mirror-test.export /tmp/tmp.TfDn6BcQ3N/cluster1-mirror-test.export 2023-08-26T13:42:26.659 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rbd --cluster cluster2 -p mirror export test@snap9-clean /tmp/tmp.TfDn6BcQ3N/cluster2-mirror-test.export 2023-08-26T13:42:27.264 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2023-08-26T13:42:27.275 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rbd --cluster cluster1 -p mirror export test@snap9-clean /tmp/tmp.TfDn6BcQ3N/cluster1-mirror-test.export 2023-08-26T13:42:27.980 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2023-08-26T13:42:27.988 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ cmp /tmp/tmp.TfDn6BcQ3N/cluster2-mirror-test.export /tmp/tmp.TfDn6BcQ3N/cluster1-mirror-test.export 2023-08-26T13:42:28.198 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rm -f /tmp/tmp.TfDn6BcQ3N/cluster2-mirror-test.export /tmp/tmp.TfDn6BcQ3N/cluster1-mirror-test.export 2023-08-26T13:42:28.345 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ return 0 2023-08-26T13:42:28.345 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ for i in `seq 1 10` 2023-08-26T13:42:28.345 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ snap_name=snap10 2023-08-26T13:42:28.345 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ compare_image_snaps mirror test snap10 2023-08-26T13:42:28.346 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T13:42:28.346 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=test 2023-08-26T13:42:28.346 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local snap_name=snap10 2023-08-26T13:42:28.346 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local ret=0 2023-08-26T13:42:28.346 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local rmt_export=/tmp/tmp.TfDn6BcQ3N/cluster2-mirror-test.export 2023-08-26T13:42:28.346 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local loc_export=/tmp/tmp.TfDn6BcQ3N/cluster1-mirror-test.export 2023-08-26T13:42:28.347 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rm -f /tmp/tmp.TfDn6BcQ3N/cluster2-mirror-test.export /tmp/tmp.TfDn6BcQ3N/cluster1-mirror-test.export 2023-08-26T13:42:28.347 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rbd --cluster cluster2 -p mirror export test@snap10 /tmp/tmp.TfDn6BcQ3N/cluster2-mirror-test.export 2023-08-26T13:42:28.926 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2023-08-26T13:42:28.936 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rbd --cluster cluster1 -p mirror export test@snap10 /tmp/tmp.TfDn6BcQ3N/cluster1-mirror-test.export 2023-08-26T13:42:29.667 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2023-08-26T13:42:29.675 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ cmp /tmp/tmp.TfDn6BcQ3N/cluster2-mirror-test.export /tmp/tmp.TfDn6BcQ3N/cluster1-mirror-test.export 2023-08-26T13:42:29.888 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rm -f /tmp/tmp.TfDn6BcQ3N/cluster2-mirror-test.export /tmp/tmp.TfDn6BcQ3N/cluster1-mirror-test.export 2023-08-26T13:42:30.032 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ return 0 2023-08-26T13:42:30.032 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ snap_name=snap10-clean 2023-08-26T13:42:30.033 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ compare_image_snaps mirror test snap10-clean 2023-08-26T13:42:30.033 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T13:42:30.033 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=test 2023-08-26T13:42:30.033 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local snap_name=snap10-clean 2023-08-26T13:42:30.033 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local ret=0 2023-08-26T13:42:30.034 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local rmt_export=/tmp/tmp.TfDn6BcQ3N/cluster2-mirror-test.export 2023-08-26T13:42:30.034 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local loc_export=/tmp/tmp.TfDn6BcQ3N/cluster1-mirror-test.export 2023-08-26T13:42:30.034 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rm -f /tmp/tmp.TfDn6BcQ3N/cluster2-mirror-test.export /tmp/tmp.TfDn6BcQ3N/cluster1-mirror-test.export 2023-08-26T13:42:30.034 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rbd --cluster cluster2 -p mirror export test@snap10-clean /tmp/tmp.TfDn6BcQ3N/cluster2-mirror-test.export 2023-08-26T13:42:30.591 INFO:tasks.rbd_mirror_thrash:waiting for cluster1.client.mirror.0 2023-08-26T13:42:30.592 INFO:teuthology.orchestra.run:waiting for 600 2023-08-26T13:42:30.592 INFO:tasks.rbd_mirror_thrash:reviving cluster1.client.mirror.0 2023-08-26T13:42:30.592 INFO:tasks.rbd_mirror.cluster1.client.mirror.0:Restarting daemon 2023-08-26T13:42:30.592 DEBUG:teuthology.orchestra.run.smithi124:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.0 2023-08-26T13:42:30.595 INFO:tasks.rbd_mirror.cluster1.client.mirror.0:Started 2023-08-26T13:42:30.595 INFO:tasks.rbd_mirror_thrash:waiting for 82 secs before thrashing 2023-08-26T13:42:30.621 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2023-08-26T13:42:30.632 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rbd --cluster cluster1 -p mirror export test@snap10-clean /tmp/tmp.TfDn6BcQ3N/cluster1-mirror-test.export 2023-08-26T13:42:31.449 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2023-08-26T13:42:31.457 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ cmp /tmp/tmp.TfDn6BcQ3N/cluster2-mirror-test.export /tmp/tmp.TfDn6BcQ3N/cluster1-mirror-test.export 2023-08-26T13:42:31.674 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rm -f /tmp/tmp.TfDn6BcQ3N/cluster2-mirror-test.export /tmp/tmp.TfDn6BcQ3N/cluster1-mirror-test.export 2023-08-26T13:42:31.819 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ return 0 2023-08-26T13:42:31.819 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ seq 1 10 2023-08-26T13:42:31.820 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ for i in `seq 1 10` 2023-08-26T13:42:31.821 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ snap_name=snap1 2023-08-26T13:42:31.821 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ remove_snapshot cluster2 mirror test snap1 2023-08-26T13:42:31.821 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster2 2023-08-26T13:42:31.821 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T13:42:31.821 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=test 2023-08-26T13:42:31.822 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local snap=snap1 2023-08-26T13:42:31.822 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rbd --cluster cluster2 snap rm mirror/test@snap1 2023-08-26T13:42:32.027 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr: Removing snap: 100% complete...done. 2023-08-26T13:42:32.039 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ snap_name=snap1-clean 2023-08-26T13:42:32.040 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ remove_snapshot cluster2 mirror test snap1-clean 2023-08-26T13:42:32.040 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster2 2023-08-26T13:42:32.040 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T13:42:32.040 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=test 2023-08-26T13:42:32.041 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local snap=snap1-clean 2023-08-26T13:42:32.041 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rbd --cluster cluster2 snap rm mirror/test@snap1-clean 2023-08-26T13:42:33.040 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr: Removing snap: 100% complete...done. 2023-08-26T13:42:33.053 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ for i in `seq 1 10` 2023-08-26T13:42:33.054 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ snap_name=snap2 2023-08-26T13:42:33.054 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ remove_snapshot cluster2 mirror test snap2 2023-08-26T13:42:33.054 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster2 2023-08-26T13:42:33.054 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T13:42:33.055 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=test 2023-08-26T13:42:33.055 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local snap=snap2 2023-08-26T13:42:33.055 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rbd --cluster cluster2 snap rm mirror/test@snap2 2023-08-26T13:42:34.034 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr: Removing snap: 100% complete...done. 2023-08-26T13:42:34.047 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ snap_name=snap2-clean 2023-08-26T13:42:34.048 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ remove_snapshot cluster2 mirror test snap2-clean 2023-08-26T13:42:34.048 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster2 2023-08-26T13:42:34.048 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T13:42:34.048 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=test 2023-08-26T13:42:34.048 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local snap=snap2-clean 2023-08-26T13:42:34.049 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rbd --cluster cluster2 snap rm mirror/test@snap2-clean 2023-08-26T13:42:35.041 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr: Removing snap: 100% complete...done. 2023-08-26T13:42:35.066 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ for i in `seq 1 10` 2023-08-26T13:42:35.066 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ snap_name=snap3 2023-08-26T13:42:35.067 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ remove_snapshot cluster2 mirror test snap3 2023-08-26T13:42:35.067 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster2 2023-08-26T13:42:35.067 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T13:42:35.067 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=test 2023-08-26T13:42:35.067 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local snap=snap3 2023-08-26T13:42:35.068 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rbd --cluster cluster2 snap rm mirror/test@snap3 2023-08-26T13:42:36.043 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr: Removing snap: 100% complete...done. 2023-08-26T13:42:36.054 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ snap_name=snap3-clean 2023-08-26T13:42:36.054 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ remove_snapshot cluster2 mirror test snap3-clean 2023-08-26T13:42:36.054 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster2 2023-08-26T13:42:36.055 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T13:42:36.055 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=test 2023-08-26T13:42:36.055 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local snap=snap3-clean 2023-08-26T13:42:36.056 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rbd --cluster cluster2 snap rm mirror/test@snap3-clean 2023-08-26T13:42:37.047 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr: Removing snap: 100% complete...done. 2023-08-26T13:42:37.062 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ for i in `seq 1 10` 2023-08-26T13:42:37.063 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ snap_name=snap4 2023-08-26T13:42:37.063 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ remove_snapshot cluster2 mirror test snap4 2023-08-26T13:42:37.063 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster2 2023-08-26T13:42:37.063 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T13:42:37.064 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=test 2023-08-26T13:42:37.064 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local snap=snap4 2023-08-26T13:42:37.064 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rbd --cluster cluster2 snap rm mirror/test@snap4 2023-08-26T13:42:38.051 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr: Removing snap: 100% complete...done. 2023-08-26T13:42:38.065 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ snap_name=snap4-clean 2023-08-26T13:42:38.065 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ remove_snapshot cluster2 mirror test snap4-clean 2023-08-26T13:42:38.065 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster2 2023-08-26T13:42:38.065 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T13:42:38.065 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=test 2023-08-26T13:42:38.066 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local snap=snap4-clean 2023-08-26T13:42:38.066 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rbd --cluster cluster2 snap rm mirror/test@snap4-clean 2023-08-26T13:42:38.763 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr: Removing snap: 100% complete...done. 2023-08-26T13:42:38.778 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ for i in `seq 1 10` 2023-08-26T13:42:38.778 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ snap_name=snap5 2023-08-26T13:42:38.778 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ remove_snapshot cluster2 mirror test snap5 2023-08-26T13:42:38.778 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster2 2023-08-26T13:42:38.779 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T13:42:38.779 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=test 2023-08-26T13:42:38.779 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local snap=snap5 2023-08-26T13:42:38.779 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rbd --cluster cluster2 snap rm mirror/test@snap5 2023-08-26T13:42:39.758 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr: Removing snap: 100% complete...done. 2023-08-26T13:42:39.772 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ snap_name=snap5-clean 2023-08-26T13:42:39.772 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ remove_snapshot cluster2 mirror test snap5-clean 2023-08-26T13:42:39.772 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster2 2023-08-26T13:42:39.773 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T13:42:39.773 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=test 2023-08-26T13:42:39.773 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local snap=snap5-clean 2023-08-26T13:42:39.773 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rbd --cluster cluster2 snap rm mirror/test@snap5-clean 2023-08-26T13:42:40.766 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr: Removing snap: 100% complete...done. 2023-08-26T13:42:40.788 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ for i in `seq 1 10` 2023-08-26T13:42:40.789 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ snap_name=snap6 2023-08-26T13:42:40.789 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ remove_snapshot cluster2 mirror test snap6 2023-08-26T13:42:40.789 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster2 2023-08-26T13:42:40.789 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T13:42:40.790 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=test 2023-08-26T13:42:40.790 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local snap=snap6 2023-08-26T13:42:40.790 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rbd --cluster cluster2 snap rm mirror/test@snap6 2023-08-26T13:42:41.772 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr: Removing snap: 100% complete...done. 2023-08-26T13:42:41.783 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ snap_name=snap6-clean 2023-08-26T13:42:41.784 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ remove_snapshot cluster2 mirror test snap6-clean 2023-08-26T13:42:41.784 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster2 2023-08-26T13:42:41.785 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T13:42:41.785 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=test 2023-08-26T13:42:41.785 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local snap=snap6-clean 2023-08-26T13:42:41.786 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rbd --cluster cluster2 snap rm mirror/test@snap6-clean 2023-08-26T13:42:42.770 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr: Removing snap: 100% complete...done. 2023-08-26T13:42:42.785 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ for i in `seq 1 10` 2023-08-26T13:42:42.785 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ snap_name=snap7 2023-08-26T13:42:42.786 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ remove_snapshot cluster2 mirror test snap7 2023-08-26T13:42:42.786 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster2 2023-08-26T13:42:42.786 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T13:42:42.786 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=test 2023-08-26T13:42:42.786 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local snap=snap7 2023-08-26T13:42:42.787 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rbd --cluster cluster2 snap rm mirror/test@snap7 2023-08-26T13:42:43.757 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr: Removing snap: 100% complete...done. 2023-08-26T13:42:43.769 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ snap_name=snap7-clean 2023-08-26T13:42:43.770 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ remove_snapshot cluster2 mirror test snap7-clean 2023-08-26T13:42:43.770 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster2 2023-08-26T13:42:43.770 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T13:42:43.770 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=test 2023-08-26T13:42:43.770 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local snap=snap7-clean 2023-08-26T13:42:43.771 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rbd --cluster cluster2 snap rm mirror/test@snap7-clean 2023-08-26T13:42:44.785 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr: Removing snap: 100% complete...done. 2023-08-26T13:42:44.798 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ for i in `seq 1 10` 2023-08-26T13:42:44.799 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ snap_name=snap8 2023-08-26T13:42:44.799 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ remove_snapshot cluster2 mirror test snap8 2023-08-26T13:42:44.799 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster2 2023-08-26T13:42:44.799 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T13:42:44.799 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=test 2023-08-26T13:42:44.800 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local snap=snap8 2023-08-26T13:42:44.800 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rbd --cluster cluster2 snap rm mirror/test@snap8 2023-08-26T13:42:45.779 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr: Removing snap: 100% complete...done. 2023-08-26T13:42:45.792 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ snap_name=snap8-clean 2023-08-26T13:42:45.792 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ remove_snapshot cluster2 mirror test snap8-clean 2023-08-26T13:42:45.792 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster2 2023-08-26T13:42:45.792 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T13:42:45.793 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=test 2023-08-26T13:42:45.793 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local snap=snap8-clean 2023-08-26T13:42:45.793 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rbd --cluster cluster2 snap rm mirror/test@snap8-clean 2023-08-26T13:42:46.783 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr: Removing snap: 100% complete...done. 2023-08-26T13:42:46.803 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ for i in `seq 1 10` 2023-08-26T13:42:46.803 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ snap_name=snap9 2023-08-26T13:42:46.804 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ remove_snapshot cluster2 mirror test snap9 2023-08-26T13:42:46.804 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster2 2023-08-26T13:42:46.804 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T13:42:46.804 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=test 2023-08-26T13:42:46.804 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local snap=snap9 2023-08-26T13:42:46.805 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rbd --cluster cluster2 snap rm mirror/test@snap9 2023-08-26T13:42:47.789 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr: Removing snap: 100% complete...done. 2023-08-26T13:42:47.802 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ snap_name=snap9-clean 2023-08-26T13:42:47.803 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ remove_snapshot cluster2 mirror test snap9-clean 2023-08-26T13:42:47.803 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster2 2023-08-26T13:42:47.803 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T13:42:47.803 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=test 2023-08-26T13:42:47.804 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local snap=snap9-clean 2023-08-26T13:42:47.804 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rbd --cluster cluster2 snap rm mirror/test@snap9-clean 2023-08-26T13:42:48.769 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr: Removing snap: 100% complete...done. 2023-08-26T13:42:48.779 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ for i in `seq 1 10` 2023-08-26T13:42:48.780 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ snap_name=snap10 2023-08-26T13:42:48.780 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ remove_snapshot cluster2 mirror test snap10 2023-08-26T13:42:48.780 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster2 2023-08-26T13:42:48.780 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T13:42:48.781 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=test 2023-08-26T13:42:48.781 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local snap=snap10 2023-08-26T13:42:48.781 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rbd --cluster cluster2 snap rm mirror/test@snap10 2023-08-26T13:42:49.801 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr: Removing snap: 100% complete...done. 2023-08-26T13:42:49.812 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ snap_name=snap10-clean 2023-08-26T13:42:49.813 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ remove_snapshot cluster2 mirror test snap10-clean 2023-08-26T13:42:49.813 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster2 2023-08-26T13:42:49.813 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T13:42:49.813 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=test 2023-08-26T13:42:49.814 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local snap=snap10-clean 2023-08-26T13:42:49.814 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rbd --cluster cluster2 snap rm mirror/test@snap10-clean 2023-08-26T13:42:50.807 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr: Removing snap: 100% complete...done. 2023-08-26T13:42:50.819 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ remove_image_retry cluster2 mirror test 2023-08-26T13:42:50.819 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster2 2023-08-26T13:42:50.819 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T13:42:50.819 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=test 2023-08-26T13:42:50.820 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ for s in 0 1 2 4 8 16 32 2023-08-26T13:42:50.820 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ sleep 0 2023-08-26T13:42:50.821 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ remove_image cluster2 mirror test 2023-08-26T13:42:50.821 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster2 2023-08-26T13:42:50.821 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T13:42:50.821 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=test 2023-08-26T13:42:50.821 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rbd --cluster=cluster2 snap purge mirror/test 2023-08-26T13:42:50.866 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rbd --cluster=cluster2 rm mirror/test 2023-08-26T13:42:50.982 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi124.stderr:2023-08-26T13:42:50.981+0000 7f2f2947d700 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x559dfef00ea0 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2023-08-26T13:42:50.982 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi124.stderr:2023-08-26T13:42:50.981+0000 7f2f2fc8a700 -1 rbd::mirror::ImageReplayer: 0x559dfe3d2280 [3/e2f2554d-659e-4467-b7f8-dddd7ccb0ee9] operator(): start failed: (85) Interrupted system call should be restarted 2023-08-26T13:42:52.106 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr: Removing image: 1% complete... Removing image: 2% complete... Removing image: 3% complete... Removing image: 4% complete... Removing image: 5% complete... Removing image: 6% complete... Removing image: 7% complete... Removing image: 8% complete... Removing image: 9% complete... Removing image: 10% complete... Removing image: 11% complete... Removing image: 12% complete... Removing image: 13% complete... Removing image: 14% complete... Removing image: 15% complete... Removing image: 16% complete... Removing image: 17% complete... Removing image: 18% complete... Removing image: 19% complete... Removing image: 20% complete... Removing image: 21% complete... Removing image: 22% complete... Removing image: 23% complete... Removing image: 24% complete... Removing image: 25% complete... Removing image: 26% complete... Removing image: 27% complete... Removing image: 28% complete... Removing image: 29% complete... Removing image: 30% complete... Removing image: 31% complete... Removing image: 32% complete... Removing image: 33% complete... Removing image: 34% complete... Removing image: 35% complete... Removing image: 36% complete... Removing image: 37% complete... Removing image: 38% complete... Removing image: 39% complete... Removing image: 40% complete... Removing image: 41% complete... Removing image: 42% complete... Removing image: 43% complete... Removing image: 44% complete... Removing image: 45% complete... Removing image: 46% complete... Removing image: 47% complete... Removing image: 48% complete... Removing image: 49% complete... Removing image: 50% complete... Removing image: 51% complete... Removing image: 52% complete... Removing image: 53% complete... Removing image: 54% complete... Removing image: 55% complete... Removing image: 56% complete... Removing image: 57% complete... Removing image: 58% complete... Removing image: 59% complete... Removing image: 60% complete... Removing image: 61% complete... Removing image: 62% complete... Removing image: 63% complete... Removing image: 64% complete... Removing image: 65% complete... Removing image: 66% complete... Removing image: 67% complete... Removing image: 68% complete... Removing image: 69% complete... Removing image: 70% complete... Removing image: 71% complete... Removing image: 72% complete... Removing image: 73% complete... Removing image: 74% complete... Removing image: 75% complete... Removing image: 76% complete... Removing image: 77% complete... Removing image: 78% complete... Removing image: 79% complete... Removing image: 80% complete... Removing image: 81% complete... Removing image: 82% complete... Removing image: 83% complete... Removing image: 84% complete... Removing image: 85% complete... Removing image: 86% complete... Removing image: 87% complete... Removing image: 88% complete... Removing image: 89% complete... Removing image: 90% complete... Removing image: 91% complete... Removing image: 92% complete... Removing image: 93% complete... Removing image: 94% complete... Removing image: 95% complete... Removing image: 96% complete... Removing image: 97% complete... Removing image: 98% complete... Removing image: 99% complete... Removing image: 100% complete...done. 2023-08-26T13:42:52.110 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ return 0 2023-08-26T13:42:52.110 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ wait_for_image_present cluster1 mirror test deleted 2023-08-26T13:42:52.110 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster1 2023-08-26T13:42:52.111 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T13:42:52.111 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=test 2023-08-26T13:42:52.111 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local state=deleted 2023-08-26T13:42:52.111 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image_id= 2023-08-26T13:42:52.112 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local s 2023-08-26T13:42:52.112 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ test -n '' 2023-08-26T13:42:52.112 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ get_image_id cluster1 mirror test 2023-08-26T13:42:52.112 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local cluster=cluster1 2023-08-26T13:42:52.112 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local pool=mirror 2023-08-26T13:42:52.112 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local image=test 2023-08-26T13:42:52.113 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ rbd --cluster=cluster1 info mirror/test 2023-08-26T13:42:52.113 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2023-08-26T13:42:52.154 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:rbd: error opening image test: (2) No such file or directory 2023-08-26T13:42:52.157 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ image_id= 2023-08-26T13:42:52.157 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2023-08-26T13:42:52.157 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ sleep 0.1 2023-08-26T13:42:52.258 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ test_image_present cluster1 mirror test deleted '' 2023-08-26T13:42:52.259 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster1 2023-08-26T13:42:52.259 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T13:42:52.259 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=test 2023-08-26T13:42:52.259 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local test_state=deleted 2023-08-26T13:42:52.260 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image_id= 2023-08-26T13:42:52.260 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local current_state=deleted 2023-08-26T13:42:52.260 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local current_image_id 2023-08-26T13:42:52.260 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ get_image_id cluster1 mirror test 2023-08-26T13:42:52.260 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local cluster=cluster1 2023-08-26T13:42:52.260 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local pool=mirror 2023-08-26T13:42:52.261 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local image=test 2023-08-26T13:42:52.261 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ rbd --cluster=cluster1 info mirror/test 2023-08-26T13:42:52.261 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2023-08-26T13:42:52.291 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:rbd: error opening image test: (2) No such file or directory 2023-08-26T13:42:52.294 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ current_image_id= 2023-08-26T13:42:52.294 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ test -n '' 2023-08-26T13:42:52.294 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ test deleted = deleted 2023-08-26T13:42:52.295 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ return 0 2023-08-26T13:42:52.295 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ testlog 'TEST: create many images' 2023-08-26T13:42:52.295 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ tee -a /tmp/tmp.TfDn6BcQ3N/rbd-mirror.test.log 2023-08-26T13:42:52.295 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ date '+%F %T' 2023-08-26T13:42:52.297 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ echo 2023-08-26 13:42:52 TEST: create many images 2023-08-26T13:42:52.297 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:2023-08-26 13:42:52 TEST: create many images 2023-08-26T13:42:52.298 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ snap_name=snap 2023-08-26T13:42:52.298 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ seq 1 50 2023-08-26T13:42:52.298 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-08-26T13:42:52.298 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ image=image_1 2023-08-26T13:42:52.298 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ create_image_and_enable_mirror cluster2 mirror image_1 snapshot 128M 2023-08-26T13:42:52.299 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster2 2023-08-26T13:42:52.299 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ shift 2023-08-26T13:42:52.299 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T13:42:52.299 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ shift 2023-08-26T13:42:52.299 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=image_1 2023-08-26T13:42:52.300 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ shift 2023-08-26T13:42:52.300 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local mode=snapshot 2023-08-26T13:42:52.300 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ '[' -n snapshot ']' 2023-08-26T13:42:52.300 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ shift 2023-08-26T13:42:52.300 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ create_image cluster2 mirror image_1 128M 2023-08-26T13:42:52.300 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster2 2023-08-26T13:42:52.301 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ shift 2023-08-26T13:42:52.301 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T13:42:52.301 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ shift 2023-08-26T13:42:52.301 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=image_1 2023-08-26T13:42:52.301 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ shift 2023-08-26T13:42:52.301 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local size=128 2023-08-26T13:42:52.302 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ '[' -n 128M ']' 2023-08-26T13:42:52.302 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ size=128M 2023-08-26T13:42:52.302 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ shift 2023-08-26T13:42:52.302 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock mirror/image_1 2023-08-26T13:42:52.344 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ '[' image = image ']' 2023-08-26T13:42:52.344 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ enable_mirror cluster2 mirror image_1 snapshot 2023-08-26T13:42:52.345 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster2 2023-08-26T13:42:52.345 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T13:42:52.345 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=image_1 2023-08-26T13:42:52.345 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local mode=snapshot 2023-08-26T13:42:52.345 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_1 snapshot 2023-08-26T13:42:52.821 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:Mirroring enabled 2023-08-26T13:42:52.827 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rbd --cluster=cluster2 info mirror/image_1 2023-08-26T13:42:52.863 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:rbd image 'image_1': 2023-08-26T13:42:52.863 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: size 128 MiB in 32 objects 2023-08-26T13:42:52.863 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: order 22 (4 MiB objects) 2023-08-26T13:42:52.864 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: snapshot_count: 1 2023-08-26T13:42:52.864 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: id: 1275fd87b672 2023-08-26T13:42:52.864 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: block_name_prefix: rbd_data.1275fd87b672 2023-08-26T13:42:52.864 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: format: 2 2023-08-26T13:42:52.864 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: features: layering, exclusive-lock 2023-08-26T13:42:52.865 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: op_features: 2023-08-26T13:42:52.865 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: flags: 2023-08-26T13:42:52.865 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: create_timestamp: Sat Aug 26 13:42:52 2023 2023-08-26T13:42:52.865 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: access_timestamp: Sat Aug 26 13:42:52 2023 2023-08-26T13:42:52.865 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: modify_timestamp: Sat Aug 26 13:42:52 2023 2023-08-26T13:42:52.866 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: mirroring state: enabled 2023-08-26T13:42:52.866 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: mirroring mode: snapshot 2023-08-26T13:42:52.866 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: mirroring global id: cab3ca63-f126-4865-b2a3-b025783f5e8a 2023-08-26T13:42:52.866 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: mirroring primary: true 2023-08-26T13:42:52.867 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ '[' -n '' ']' 2023-08-26T13:42:52.867 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ stress_write_image cluster2 mirror image_1 2023-08-26T13:42:52.867 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster2 2023-08-26T13:42:52.868 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T13:42:52.868 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=image_1 2023-08-26T13:42:52.868 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2023-08-26T13:42:52.869 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local duration=11 2023-08-26T13:42:52.869 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ set +e 2023-08-26T13:42:52.869 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ timeout 11s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_1 --debug-rbd=20 --debug-journaler=20 2023-08-26T13:42:53.057 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:100 IOs, 30684160 bytes 2023-08-26T13:42:53.141 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:200 IOs, 31892480 bytes 2023-08-26T13:42:53.284 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:300 IOs, 69454848 bytes 2023-08-26T13:42:53.409 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:400 IOs, 94895104 bytes 2023-08-26T13:42:53.534 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:500 IOs, 120567808 bytes 2023-08-26T13:42:53.646 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:600 IOs, 135532544 bytes 2023-08-26T13:42:53.765 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:700 IOs, 149025792 bytes 2023-08-26T13:42:53.975 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:800 IOs, 220985344 bytes 2023-08-26T13:42:54.121 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:900 IOs, 254895104 bytes 2023-08-26T13:42:54.214 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:1000 IOs, 257698816 bytes 2023-08-26T13:42:54.389 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:1100 IOs, 311077888 bytes 2023-08-26T13:42:54.542 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:1200 IOs, 348226560 bytes 2023-08-26T13:42:54.695 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:1300 IOs, 384855040 bytes 2023-08-26T13:42:54.848 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:1400 IOs, 446991360 bytes 2023-08-26T13:42:55.095 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:1500 IOs, 517452800 bytes 2023-08-26T13:42:55.216 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:1600 IOs, 539476992 bytes 2023-08-26T13:42:55.350 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:1700 IOs, 579898368 bytes 2023-08-26T13:42:55.513 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:1800 IOs, 611336192 bytes 2023-08-26T13:42:55.750 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:1900 IOs, 717757440 bytes 2023-08-26T13:42:55.930 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:2000 IOs, 738660352 bytes 2023-08-26T13:42:56.053 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:2100 IOs, 776551424 bytes 2023-08-26T13:42:56.283 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:2200 IOs, 847436800 bytes 2023-08-26T13:42:56.417 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:2300 IOs, 863657984 bytes 2023-08-26T13:42:56.625 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:2400 IOs, 922978304 bytes 2023-08-26T13:42:56.811 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:2500 IOs, 971577344 bytes 2023-08-26T13:42:56.984 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:2600 IOs, 1014875136 bytes 2023-08-26T13:42:57.163 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:2700 IOs, 1066200064 bytes 2023-08-26T13:42:57.265 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:2800 IOs, 1088789504 bytes 2023-08-26T13:42:57.410 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:2900 IOs, 1114727424 bytes 2023-08-26T13:42:57.537 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:3000 IOs, 1138515968 bytes 2023-08-26T13:42:57.679 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:3100 IOs, 1186010112 bytes 2023-08-26T13:42:57.861 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:3200 IOs, 1228566528 bytes 2023-08-26T13:42:57.975 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:3300 IOs, 1245295616 bytes 2023-08-26T13:42:58.110 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:3400 IOs, 1267016704 bytes 2023-08-26T13:42:58.212 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:3500 IOs, 1276218368 bytes 2023-08-26T13:42:58.340 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:3600 IOs, 1300404224 bytes 2023-08-26T13:42:58.422 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:3700 IOs, 1301040128 bytes 2023-08-26T13:42:58.718 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:3800 IOs, 1409588224 bytes 2023-08-26T13:42:58.809 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:3900 IOs, 1410204672 bytes 2023-08-26T13:42:58.917 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:4000 IOs, 1428061184 bytes 2023-08-26T13:42:58.998 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:4100 IOs, 1428676608 bytes 2023-08-26T13:42:59.186 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:4200 IOs, 1472290816 bytes 2023-08-26T13:42:59.340 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:4300 IOs, 1511120896 bytes 2023-08-26T13:42:59.432 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:4400 IOs, 1524914176 bytes 2023-08-26T13:42:59.573 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:4500 IOs, 1554628608 bytes 2023-08-26T13:42:59.655 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:4600 IOs, 1555250176 bytes 2023-08-26T13:42:59.858 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:4700 IOs, 1616687104 bytes 2023-08-26T13:43:00.086 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:4800 IOs, 1681435648 bytes 2023-08-26T13:43:00.245 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:4900 IOs, 1719612416 bytes 2023-08-26T13:43:00.403 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:5000 IOs, 1761558528 bytes 2023-08-26T13:43:00.519 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:5100 IOs, 1778349056 bytes 2023-08-26T13:43:00.645 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:5200 IOs, 1803724800 bytes 2023-08-26T13:43:00.919 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:5300 IOs, 1897312256 bytes 2023-08-26T13:43:01.102 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:5400 IOs, 1960183808 bytes 2023-08-26T13:43:01.281 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:5500 IOs, 1982540800 bytes 2023-08-26T13:43:01.485 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:5600 IOs, 2046884864 bytes 2023-08-26T13:43:01.614 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:5700 IOs, 2065214464 bytes 2023-08-26T13:43:01.819 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:5800 IOs, 2116630528 bytes 2023-08-26T13:43:01.986 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:5900 IOs, 2161155072 bytes 2023-08-26T13:43:02.160 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:6000 IOs, 2218261504 bytes 2023-08-26T13:43:02.347 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:6100 IOs, 2263931904 bytes 2023-08-26T13:43:02.484 INFO:tasks.rbd_mirror_thrash:kill cluster2.client.mirror.0 2023-08-26T13:43:02.485 INFO:tasks.rbd_mirror.cluster2.client.mirror.0:Sent signal 15 2023-08-26T13:43:02.485 INFO:tasks.rbd_mirror_thrash:waiting for 2 secs before reviving daemons 2023-08-26T13:43:02.486 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi124.stderr:2023-08-26T13:43:02.484+0000 7f0413a7e700 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.0 (PID: 112877) UID: 1000 2023-08-26T13:43:02.516 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:6200 IOs, 2293338112 bytes 2023-08-26T13:43:02.710 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:6300 IOs, 2344205312 bytes 2023-08-26T13:43:02.947 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:6400 IOs, 2427827200 bytes 2023-08-26T13:43:03.046 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:6500 IOs, 2428438528 bytes 2023-08-26T13:43:03.246 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:6600 IOs, 2492217344 bytes 2023-08-26T13:43:03.408 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:6700 IOs, 2533246976 bytes 2023-08-26T13:43:03.539 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:6800 IOs, 2558027776 bytes 2023-08-26T13:43:03.670 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:6900 IOs, 2595611648 bytes 2023-08-26T13:43:03.874 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ error_code=124 2023-08-26T13:43:03.874 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ set -e 2023-08-26T13:43:03.874 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ '[' 124 -eq 124 ']' 2023-08-26T13:43:03.875 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ return 0 2023-08-26T13:43:03.875 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-08-26T13:43:03.875 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ image=image_2 2023-08-26T13:43:03.875 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ create_image_and_enable_mirror cluster2 mirror image_2 snapshot 128M 2023-08-26T13:43:03.876 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster2 2023-08-26T13:43:03.876 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ shift 2023-08-26T13:43:03.876 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T13:43:03.876 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ shift 2023-08-26T13:43:03.876 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=image_2 2023-08-26T13:43:03.877 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ shift 2023-08-26T13:43:03.877 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local mode=snapshot 2023-08-26T13:43:03.877 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ '[' -n snapshot ']' 2023-08-26T13:43:03.877 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ shift 2023-08-26T13:43:03.877 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ create_image cluster2 mirror image_2 128M 2023-08-26T13:43:03.877 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster2 2023-08-26T13:43:03.878 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ shift 2023-08-26T13:43:03.878 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T13:43:03.878 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ shift 2023-08-26T13:43:03.878 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=image_2 2023-08-26T13:43:03.878 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ shift 2023-08-26T13:43:03.879 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local size=128 2023-08-26T13:43:03.879 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ '[' -n 128M ']' 2023-08-26T13:43:03.879 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ size=128M 2023-08-26T13:43:03.879 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ shift 2023-08-26T13:43:03.879 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock mirror/image_2 2023-08-26T13:43:03.917 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ '[' image = image ']' 2023-08-26T13:43:03.918 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ enable_mirror cluster2 mirror image_2 snapshot 2023-08-26T13:43:03.918 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster2 2023-08-26T13:43:03.918 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T13:43:03.919 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=image_2 2023-08-26T13:43:03.919 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local mode=snapshot 2023-08-26T13:43:03.919 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_2 snapshot 2023-08-26T13:43:04.486 INFO:tasks.rbd_mirror_thrash:waiting for cluster2.client.mirror.0 2023-08-26T13:43:04.486 INFO:teuthology.orchestra.run:waiting for 600 2023-08-26T13:43:04.486 INFO:tasks.rbd_mirror_thrash:reviving cluster2.client.mirror.0 2023-08-26T13:43:04.487 INFO:tasks.rbd_mirror.cluster2.client.mirror.0:Restarting daemon 2023-08-26T13:43:04.487 DEBUG:teuthology.orchestra.run.smithi124:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.0 2023-08-26T13:43:04.490 INFO:tasks.rbd_mirror.cluster2.client.mirror.0:Started 2023-08-26T13:43:04.490 INFO:tasks.rbd_mirror_thrash:waiting for 70 secs before thrashing 2023-08-26T13:43:04.637 INFO:tasks.ceph.cluster2.osd.2.smithi124.stderr:2023-08-26T13:43:04.636+0000 7fb790854700 -1 reset not still connected to 0x5598f06d2f70 2023-08-26T13:43:04.638 INFO:tasks.ceph.cluster2.osd.2.smithi124.stderr:2023-08-26T13:43:04.637+0000 7fb790854700 -1 reset not still connected to 0x5598f06d31e0 2023-08-26T13:43:04.771 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:Mirroring enabled 2023-08-26T13:43:04.778 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rbd --cluster=cluster2 info mirror/image_2 2023-08-26T13:43:04.814 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:rbd image 'image_2': 2023-08-26T13:43:04.814 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: size 128 MiB in 32 objects 2023-08-26T13:43:04.814 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: order 22 (4 MiB objects) 2023-08-26T13:43:04.814 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: snapshot_count: 1 2023-08-26T13:43:04.815 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: id: 1280bc6de3f7 2023-08-26T13:43:04.815 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: block_name_prefix: rbd_data.1280bc6de3f7 2023-08-26T13:43:04.815 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: format: 2 2023-08-26T13:43:04.815 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: features: layering, exclusive-lock 2023-08-26T13:43:04.815 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: op_features: 2023-08-26T13:43:04.816 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: flags: 2023-08-26T13:43:04.816 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: create_timestamp: Sat Aug 26 13:43:03 2023 2023-08-26T13:43:04.816 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: access_timestamp: Sat Aug 26 13:43:03 2023 2023-08-26T13:43:04.816 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: modify_timestamp: Sat Aug 26 13:43:03 2023 2023-08-26T13:43:04.816 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: mirroring state: enabled 2023-08-26T13:43:04.817 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: mirroring mode: snapshot 2023-08-26T13:43:04.817 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: mirroring global id: 072383ad-81e1-4717-a595-674819d66895 2023-08-26T13:43:04.817 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: mirroring primary: true 2023-08-26T13:43:04.818 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ '[' -n '' ']' 2023-08-26T13:43:04.818 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ stress_write_image cluster2 mirror image_2 2023-08-26T13:43:04.818 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster2 2023-08-26T13:43:04.819 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T13:43:04.819 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=image_2 2023-08-26T13:43:04.819 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2023-08-26T13:43:04.820 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local duration=5 2023-08-26T13:43:04.820 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ set +e 2023-08-26T13:43:04.820 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ timeout 5s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_2 --debug-rbd=20 --debug-journaler=20 2023-08-26T13:43:04.963 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:100 IOs, 11901952 bytes 2023-08-26T13:43:05.144 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:200 IOs, 56960000 bytes 2023-08-26T13:43:05.308 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:300 IOs, 100756480 bytes 2023-08-26T13:43:05.465 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:400 IOs, 138752000 bytes 2023-08-26T13:43:05.583 INFO:tasks.ceph.cluster2.osd.1.smithi124.stderr:2023-08-26T13:43:05.582+0000 7f2a5dd89700 -1 reset not still connected to 0x5626611735f0 2023-08-26T13:43:05.705 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:500 IOs, 216155136 bytes 2023-08-26T13:43:05.857 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:600 IOs, 255737856 bytes 2023-08-26T13:43:06.033 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:700 IOs, 292203520 bytes 2023-08-26T13:43:06.117 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:800 IOs, 292834304 bytes 2023-08-26T13:43:06.322 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:900 IOs, 352405504 bytes 2023-08-26T13:43:06.461 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:1000 IOs, 380717056 bytes 2023-08-26T13:43:06.563 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:1100 IOs, 386240512 bytes 2023-08-26T13:43:06.709 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:1200 IOs, 415412224 bytes 2023-08-26T13:43:06.825 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:1300 IOs, 431474688 bytes 2023-08-26T13:43:06.969 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:1400 IOs, 464841728 bytes 2023-08-26T13:43:07.241 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:1500 IOs, 557422592 bytes 2023-08-26T13:43:07.367 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:1600 IOs, 581682176 bytes 2023-08-26T13:43:07.649 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:1700 IOs, 683376640 bytes 2023-08-26T13:43:07.829 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:1800 IOs, 737146880 bytes 2023-08-26T13:43:08.064 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:1900 IOs, 809817088 bytes 2023-08-26T13:43:08.222 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:2000 IOs, 845338624 bytes 2023-08-26T13:43:08.373 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:2100 IOs, 883651584 bytes 2023-08-26T13:43:08.525 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:2200 IOs, 912972800 bytes 2023-08-26T13:43:08.644 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:2300 IOs, 936024064 bytes 2023-08-26T13:43:08.910 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:2400 IOs, 1033853952 bytes 2023-08-26T13:43:09.116 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:2500 IOs, 1079185408 bytes 2023-08-26T13:43:09.252 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:2600 IOs, 1104986112 bytes 2023-08-26T13:43:09.363 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:2700 IOs, 1114387456 bytes 2023-08-26T13:43:09.512 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:2800 IOs, 1151133696 bytes 2023-08-26T13:43:09.741 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:2900 IOs, 1220631552 bytes 2023-08-26T13:43:09.823 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ error_code=124 2023-08-26T13:43:09.824 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ set -e 2023-08-26T13:43:09.824 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ '[' 124 -eq 124 ']' 2023-08-26T13:43:09.824 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ return 0 2023-08-26T13:43:09.824 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-08-26T13:43:09.825 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ image=image_3 2023-08-26T13:43:09.825 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ create_image_and_enable_mirror cluster2 mirror image_3 snapshot 128M 2023-08-26T13:43:09.825 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster2 2023-08-26T13:43:09.825 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ shift 2023-08-26T13:43:09.825 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T13:43:09.826 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ shift 2023-08-26T13:43:09.826 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=image_3 2023-08-26T13:43:09.826 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ shift 2023-08-26T13:43:09.826 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local mode=snapshot 2023-08-26T13:43:09.826 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ '[' -n snapshot ']' 2023-08-26T13:43:09.827 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ shift 2023-08-26T13:43:09.827 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ create_image cluster2 mirror image_3 128M 2023-08-26T13:43:09.827 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster2 2023-08-26T13:43:09.827 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ shift 2023-08-26T13:43:09.827 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T13:43:09.827 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ shift 2023-08-26T13:43:09.828 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=image_3 2023-08-26T13:43:09.828 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ shift 2023-08-26T13:43:09.828 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local size=128 2023-08-26T13:43:09.828 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ '[' -n 128M ']' 2023-08-26T13:43:09.828 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ size=128M 2023-08-26T13:43:09.829 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ shift 2023-08-26T13:43:09.829 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock mirror/image_3 2023-08-26T13:43:09.867 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ '[' image = image ']' 2023-08-26T13:43:09.867 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ enable_mirror cluster2 mirror image_3 snapshot 2023-08-26T13:43:09.867 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster2 2023-08-26T13:43:09.867 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T13:43:09.867 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=image_3 2023-08-26T13:43:09.868 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local mode=snapshot 2023-08-26T13:43:09.868 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_3 snapshot 2023-08-26T13:43:10.083 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:Mirroring enabled 2023-08-26T13:43:10.089 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rbd --cluster=cluster2 info mirror/image_3 2023-08-26T13:43:10.125 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:rbd image 'image_3': 2023-08-26T13:43:10.125 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: size 128 MiB in 32 objects 2023-08-26T13:43:10.125 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: order 22 (4 MiB objects) 2023-08-26T13:43:10.125 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: snapshot_count: 1 2023-08-26T13:43:10.125 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: id: 129262d2c055 2023-08-26T13:43:10.126 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: block_name_prefix: rbd_data.129262d2c055 2023-08-26T13:43:10.126 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: format: 2 2023-08-26T13:43:10.126 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: features: layering, exclusive-lock 2023-08-26T13:43:10.126 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: op_features: 2023-08-26T13:43:10.126 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: flags: 2023-08-26T13:43:10.127 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: create_timestamp: Sat Aug 26 13:43:09 2023 2023-08-26T13:43:10.127 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: access_timestamp: Sat Aug 26 13:43:09 2023 2023-08-26T13:43:10.127 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: modify_timestamp: Sat Aug 26 13:43:09 2023 2023-08-26T13:43:10.128 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: mirroring state: enabled 2023-08-26T13:43:10.128 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: mirroring mode: snapshot 2023-08-26T13:43:10.128 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: mirroring global id: 339416f3-5c35-427b-8a40-65d1d209d742 2023-08-26T13:43:10.128 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: mirroring primary: true 2023-08-26T13:43:10.130 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ '[' -n '' ']' 2023-08-26T13:43:10.130 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ stress_write_image cluster2 mirror image_3 2023-08-26T13:43:10.130 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster2 2023-08-26T13:43:10.130 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T13:43:10.130 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=image_3 2023-08-26T13:43:10.131 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2023-08-26T13:43:10.131 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local duration=6 2023-08-26T13:43:10.131 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ set +e 2023-08-26T13:43:10.131 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ timeout 6s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_3 --debug-rbd=20 --debug-journaler=20 2023-08-26T13:43:10.307 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:100 IOs, 32966656 bytes 2023-08-26T13:43:10.517 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:200 IOs, 105970688 bytes 2023-08-26T13:43:10.683 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:300 IOs, 140059648 bytes 2023-08-26T13:43:10.875 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:400 IOs, 200129536 bytes 2023-08-26T13:43:10.986 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:500 IOs, 217766912 bytes 2023-08-26T13:43:11.173 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:600 IOs, 268072960 bytes 2023-08-26T13:43:11.396 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:700 IOs, 340538368 bytes 2023-08-26T13:43:11.579 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:800 IOs, 385428480 bytes 2023-08-26T13:43:11.698 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:900 IOs, 406549504 bytes 2023-08-26T13:43:11.917 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:1000 IOs, 471703552 bytes 2023-08-26T13:43:12.145 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:1100 IOs, 552203264 bytes 2023-08-26T13:43:12.292 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:1200 IOs, 569701376 bytes 2023-08-26T13:43:12.429 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:1300 IOs, 600434688 bytes 2023-08-26T13:43:12.566 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:1400 IOs, 631758848 bytes 2023-08-26T13:43:12.782 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:1500 IOs, 699010048 bytes 2023-08-26T13:43:13.005 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:1600 IOs, 769360896 bytes 2023-08-26T13:43:13.139 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:1700 IOs, 796630016 bytes 2023-08-26T13:43:13.251 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:1800 IOs, 799451136 bytes 2023-08-26T13:43:13.341 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:1900 IOs, 825905152 bytes 2023-08-26T13:43:13.486 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:2000 IOs, 853811200 bytes 2023-08-26T13:43:13.760 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:2100 IOs, 945179648 bytes 2023-08-26T13:43:13.879 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:2200 IOs, 961774592 bytes 2023-08-26T13:43:14.002 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:2300 IOs, 975455232 bytes 2023-08-26T13:43:14.229 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:2400 IOs, 1067244544 bytes 2023-08-26T13:43:14.437 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:2500 IOs, 1107299328 bytes 2023-08-26T13:43:14.577 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:2600 IOs, 1133534208 bytes 2023-08-26T13:43:14.766 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:2700 IOs, 1186049024 bytes 2023-08-26T13:43:14.958 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:2800 IOs, 1233348608 bytes 2023-08-26T13:43:15.088 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:2900 IOs, 1260426240 bytes 2023-08-26T13:43:15.306 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:3000 IOs, 1354464256 bytes 2023-08-26T13:43:15.449 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:3100 IOs, 1368186880 bytes 2023-08-26T13:43:15.647 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:3200 IOs, 1413262336 bytes 2023-08-26T13:43:15.839 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:3300 IOs, 1469267968 bytes 2023-08-26T13:43:15.973 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:3400 IOs, 1499936768 bytes 2023-08-26T13:43:16.137 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ error_code=124 2023-08-26T13:43:16.137 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ set -e 2023-08-26T13:43:16.138 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ '[' 124 -eq 124 ']' 2023-08-26T13:43:16.138 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ return 0 2023-08-26T13:43:16.138 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-08-26T13:43:16.138 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ image=image_4 2023-08-26T13:43:16.138 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ create_image_and_enable_mirror cluster2 mirror image_4 snapshot 128M 2023-08-26T13:43:16.139 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster2 2023-08-26T13:43:16.139 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ shift 2023-08-26T13:43:16.139 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T13:43:16.139 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ shift 2023-08-26T13:43:16.139 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=image_4 2023-08-26T13:43:16.140 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ shift 2023-08-26T13:43:16.140 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local mode=snapshot 2023-08-26T13:43:16.140 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ '[' -n snapshot ']' 2023-08-26T13:43:16.140 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ shift 2023-08-26T13:43:16.140 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ create_image cluster2 mirror image_4 128M 2023-08-26T13:43:16.141 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster2 2023-08-26T13:43:16.141 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ shift 2023-08-26T13:43:16.141 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T13:43:16.141 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ shift 2023-08-26T13:43:16.141 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=image_4 2023-08-26T13:43:16.142 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ shift 2023-08-26T13:43:16.142 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local size=128 2023-08-26T13:43:16.142 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ '[' -n 128M ']' 2023-08-26T13:43:16.142 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ size=128M 2023-08-26T13:43:16.142 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ shift 2023-08-26T13:43:16.143 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock mirror/image_4 2023-08-26T13:43:16.180 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ '[' image = image ']' 2023-08-26T13:43:16.180 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ enable_mirror cluster2 mirror image_4 snapshot 2023-08-26T13:43:16.180 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster2 2023-08-26T13:43:16.181 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T13:43:16.181 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=image_4 2023-08-26T13:43:16.181 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local mode=snapshot 2023-08-26T13:43:16.181 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_4 snapshot 2023-08-26T13:43:16.804 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:Mirroring enabled 2023-08-26T13:43:16.811 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rbd --cluster=cluster2 info mirror/image_4 2023-08-26T13:43:16.849 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:rbd image 'image_4': 2023-08-26T13:43:16.849 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: size 128 MiB in 32 objects 2023-08-26T13:43:16.849 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: order 22 (4 MiB objects) 2023-08-26T13:43:16.849 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: snapshot_count: 1 2023-08-26T13:43:16.850 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: id: 129a8936c002 2023-08-26T13:43:16.850 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: block_name_prefix: rbd_data.129a8936c002 2023-08-26T13:43:16.850 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: format: 2 2023-08-26T13:43:16.850 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: features: layering, exclusive-lock 2023-08-26T13:43:16.850 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: op_features: 2023-08-26T13:43:16.851 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: flags: 2023-08-26T13:43:16.851 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: create_timestamp: Sat Aug 26 13:43:16 2023 2023-08-26T13:43:16.851 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: access_timestamp: Sat Aug 26 13:43:16 2023 2023-08-26T13:43:16.851 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: modify_timestamp: Sat Aug 26 13:43:16 2023 2023-08-26T13:43:16.851 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: mirroring state: enabled 2023-08-26T13:43:16.852 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: mirroring mode: snapshot 2023-08-26T13:43:16.852 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: mirroring global id: 2f906edf-8e50-48f7-a42e-a4055f204c44 2023-08-26T13:43:16.852 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: mirroring primary: true 2023-08-26T13:43:16.853 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ '[' -n '' ']' 2023-08-26T13:43:16.853 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ stress_write_image cluster2 mirror image_4 2023-08-26T13:43:16.853 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster2 2023-08-26T13:43:16.853 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T13:43:16.854 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=image_4 2023-08-26T13:43:16.854 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2023-08-26T13:43:16.854 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local duration=11 2023-08-26T13:43:16.855 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ set +e 2023-08-26T13:43:16.855 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ timeout 11s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_4 --debug-rbd=20 --debug-journaler=20 2023-08-26T13:43:17.130 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:100 IOs, 77957120 bytes 2023-08-26T13:43:17.222 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:200 IOs, 111525888 bytes 2023-08-26T13:43:17.480 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:300 IOs, 178663424 bytes 2023-08-26T13:43:17.656 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:400 IOs, 221718528 bytes 2023-08-26T13:43:17.809 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:500 IOs, 263640064 bytes 2023-08-26T13:43:18.004 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:600 IOs, 323463168 bytes 2023-08-26T13:43:18.214 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:700 IOs, 393533440 bytes 2023-08-26T13:43:18.305 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:800 IOs, 413094912 bytes 2023-08-26T13:43:18.547 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:900 IOs, 477300736 bytes 2023-08-26T13:43:18.740 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:1000 IOs, 536806400 bytes 2023-08-26T13:43:18.889 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:1100 IOs, 565271552 bytes 2023-08-26T13:43:19.051 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:1200 IOs, 607916032 bytes 2023-08-26T13:43:19.187 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:1300 IOs, 632583168 bytes 2023-08-26T13:43:19.326 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:1400 IOs, 663121920 bytes 2023-08-26T13:43:19.508 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:1500 IOs, 720969728 bytes 2023-08-26T13:43:19.664 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:1600 IOs, 746561536 bytes 2023-08-26T13:43:19.826 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:1700 IOs, 784057344 bytes 2023-08-26T13:43:19.990 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:1800 IOs, 830754816 bytes 2023-08-26T13:43:20.100 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:1900 IOs, 838952960 bytes 2023-08-26T13:43:20.287 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:2000 IOs, 882933760 bytes 2023-08-26T13:43:20.490 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:2100 IOs, 947301376 bytes 2023-08-26T13:43:20.584 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:2200 IOs, 953391104 bytes 2023-08-26T13:43:20.726 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:2300 IOs, 987619328 bytes 2023-08-26T13:43:20.924 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:2400 IOs, 1050146816 bytes 2023-08-26T13:43:21.167 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:2500 IOs, 1124970496 bytes 2023-08-26T13:43:21.464 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:2600 IOs, 1229316096 bytes 2023-08-26T13:43:21.763 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:2700 IOs, 1343777792 bytes 2023-08-26T13:43:21.889 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:2800 IOs, 1368434688 bytes 2023-08-26T13:43:22.046 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:2900 IOs, 1403116544 bytes 2023-08-26T13:43:22.292 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:3000 IOs, 1442591744 bytes 2023-08-26T13:43:22.474 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:3100 IOs, 1497108480 bytes 2023-08-26T13:43:22.616 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:3200 IOs, 1529068544 bytes 2023-08-26T13:43:22.900 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:3300 IOs, 1608616960 bytes 2023-08-26T13:43:23.075 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:3400 IOs, 1638707200 bytes 2023-08-26T13:43:23.399 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:3500 IOs, 1713900544 bytes 2023-08-26T13:43:23.496 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:3600 IOs, 1724703744 bytes 2023-08-26T13:43:23.658 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:3700 IOs, 1763556352 bytes 2023-08-26T13:43:23.766 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:3800 IOs, 1779137536 bytes 2023-08-26T13:43:23.980 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:3900 IOs, 1846761472 bytes 2023-08-26T13:43:24.221 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:4000 IOs, 1929120768 bytes 2023-08-26T13:43:24.434 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:4100 IOs, 1985275904 bytes 2023-08-26T13:43:24.598 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:4200 IOs, 2035204096 bytes 2023-08-26T13:43:24.735 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:4300 IOs, 2067820544 bytes 2023-08-26T13:43:24.876 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:4400 IOs, 2097233920 bytes 2023-08-26T13:43:24.965 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:4500 IOs, 2097839104 bytes 2023-08-26T13:43:25.157 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:4600 IOs, 2164603904 bytes 2023-08-26T13:43:25.339 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:4700 IOs, 2210466816 bytes 2023-08-26T13:43:25.593 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:4800 IOs, 2278685696 bytes 2023-08-26T13:43:25.796 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:4900 IOs, 2339664896 bytes 2023-08-26T13:43:25.893 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:5000 IOs, 2349118464 bytes 2023-08-26T13:43:26.014 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:5100 IOs, 2382714880 bytes 2023-08-26T13:43:26.222 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:5200 IOs, 2450851840 bytes 2023-08-26T13:43:26.347 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:5300 IOs, 2456140800 bytes 2023-08-26T13:43:26.581 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:5400 IOs, 2504292352 bytes 2023-08-26T13:43:26.784 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:5500 IOs, 2563066880 bytes 2023-08-26T13:43:26.901 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:5600 IOs, 2580514816 bytes 2023-08-26T13:43:26.996 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:5700 IOs, 2588211200 bytes 2023-08-26T13:43:27.149 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:5800 IOs, 2617580544 bytes 2023-08-26T13:43:27.357 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:5900 IOs, 2676415488 bytes 2023-08-26T13:43:27.500 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:6000 IOs, 2706969600 bytes 2023-08-26T13:43:27.665 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:6100 IOs, 2746621952 bytes 2023-08-26T13:43:27.860 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ error_code=124 2023-08-26T13:43:27.861 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ set -e 2023-08-26T13:43:27.861 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ '[' 124 -eq 124 ']' 2023-08-26T13:43:27.861 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ return 0 2023-08-26T13:43:27.861 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-08-26T13:43:27.862 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ image=image_5 2023-08-26T13:43:27.862 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ create_image_and_enable_mirror cluster2 mirror image_5 snapshot 128M 2023-08-26T13:43:27.862 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster2 2023-08-26T13:43:27.862 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ shift 2023-08-26T13:43:27.862 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T13:43:27.863 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ shift 2023-08-26T13:43:27.863 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=image_5 2023-08-26T13:43:27.863 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ shift 2023-08-26T13:43:27.863 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local mode=snapshot 2023-08-26T13:43:27.863 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ '[' -n snapshot ']' 2023-08-26T13:43:27.864 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ shift 2023-08-26T13:43:27.864 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ create_image cluster2 mirror image_5 128M 2023-08-26T13:43:27.864 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster2 2023-08-26T13:43:27.864 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ shift 2023-08-26T13:43:27.864 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T13:43:27.864 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ shift 2023-08-26T13:43:27.865 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=image_5 2023-08-26T13:43:27.865 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ shift 2023-08-26T13:43:27.865 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local size=128 2023-08-26T13:43:27.865 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ '[' -n 128M ']' 2023-08-26T13:43:27.865 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ size=128M 2023-08-26T13:43:27.866 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ shift 2023-08-26T13:43:27.866 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock mirror/image_5 2023-08-26T13:43:27.905 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ '[' image = image ']' 2023-08-26T13:43:27.906 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ enable_mirror cluster2 mirror image_5 snapshot 2023-08-26T13:43:27.906 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster2 2023-08-26T13:43:27.906 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T13:43:27.906 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=image_5 2023-08-26T13:43:27.907 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local mode=snapshot 2023-08-26T13:43:27.907 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_5 snapshot 2023-08-26T13:43:28.068 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:Mirroring enabled 2023-08-26T13:43:28.074 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rbd --cluster=cluster2 info mirror/image_5 2023-08-26T13:43:28.109 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:rbd image 'image_5': 2023-08-26T13:43:28.109 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: size 128 MiB in 32 objects 2023-08-26T13:43:28.110 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: order 22 (4 MiB objects) 2023-08-26T13:43:28.110 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: snapshot_count: 1 2023-08-26T13:43:28.110 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: id: 12a28d424b84 2023-08-26T13:43:28.110 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: block_name_prefix: rbd_data.12a28d424b84 2023-08-26T13:43:28.111 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: format: 2 2023-08-26T13:43:28.111 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: features: layering, exclusive-lock 2023-08-26T13:43:28.111 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: op_features: 2023-08-26T13:43:28.111 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: flags: 2023-08-26T13:43:28.111 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: create_timestamp: Sat Aug 26 13:43:27 2023 2023-08-26T13:43:28.112 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: access_timestamp: Sat Aug 26 13:43:27 2023 2023-08-26T13:43:28.112 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: modify_timestamp: Sat Aug 26 13:43:27 2023 2023-08-26T13:43:28.112 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: mirroring state: enabled 2023-08-26T13:43:28.112 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: mirroring mode: snapshot 2023-08-26T13:43:28.112 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: mirroring global id: fd599976-dbcc-4e1b-98dd-2440bc289607 2023-08-26T13:43:28.113 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: mirroring primary: true 2023-08-26T13:43:28.113 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ '[' -n '' ']' 2023-08-26T13:43:28.114 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ stress_write_image cluster2 mirror image_5 2023-08-26T13:43:28.114 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster2 2023-08-26T13:43:28.114 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T13:43:28.114 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=image_5 2023-08-26T13:43:28.115 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2023-08-26T13:43:28.116 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local duration=7 2023-08-26T13:43:28.116 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ set +e 2023-08-26T13:43:28.117 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ timeout 7s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_5 --debug-rbd=20 --debug-journaler=20 2023-08-26T13:43:28.321 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:100 IOs, 45451264 bytes 2023-08-26T13:43:28.590 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:200 IOs, 140259328 bytes 2023-08-26T13:43:28.687 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:300 IOs, 150014976 bytes 2023-08-26T13:43:28.863 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:400 IOs, 191628288 bytes 2023-08-26T13:43:29.016 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:500 IOs, 226677760 bytes 2023-08-26T13:43:29.138 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:600 IOs, 253115392 bytes 2023-08-26T13:43:29.289 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:700 IOs, 285596672 bytes 2023-08-26T13:43:29.451 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:800 IOs, 316352512 bytes 2023-08-26T13:43:29.706 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:900 IOs, 418313216 bytes 2023-08-26T13:43:29.873 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:1000 IOs, 454295552 bytes 2023-08-26T13:43:29.973 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:1100 IOs, 464159744 bytes 2023-08-26T13:43:30.217 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:1200 IOs, 569347072 bytes 2023-08-26T13:43:30.456 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:1300 IOs, 623649792 bytes 2023-08-26T13:43:30.658 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:1400 IOs, 679952384 bytes 2023-08-26T13:43:30.775 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:1500 IOs, 700015616 bytes 2023-08-26T13:43:30.909 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:1600 IOs, 724052992 bytes 2023-08-26T13:43:31.112 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:1700 IOs, 784626688 bytes 2023-08-26T13:43:31.320 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:1800 IOs, 846765056 bytes 2023-08-26T13:43:31.493 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:1900 IOs, 884570112 bytes 2023-08-26T13:43:31.722 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:2000 IOs, 969390080 bytes 2023-08-26T13:43:31.888 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:2100 IOs, 994469888 bytes 2023-08-26T13:43:32.025 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:2200 IOs, 1021420544 bytes 2023-08-26T13:43:32.176 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:2300 IOs, 1082589184 bytes 2023-08-26T13:43:32.338 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:2400 IOs, 1083194368 bytes 2023-08-26T13:43:32.486 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:2500 IOs, 1132589056 bytes 2023-08-26T13:43:32.643 INFO:tasks.ceph.cluster2.osd.0.smithi124.stderr:2023-08-26T13:43:32.639+0000 7fb60dfa7700 -1 reset not still connected to 0x56452e295380 2023-08-26T13:43:32.666 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:2600 IOs, 1162973184 bytes 2023-08-26T13:43:32.800 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:2700 IOs, 1191075840 bytes 2023-08-26T13:43:32.934 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:2800 IOs, 1221636096 bytes 2023-08-26T13:43:33.079 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:2900 IOs, 1259515904 bytes 2023-08-26T13:43:33.265 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:3000 IOs, 1307791360 bytes 2023-08-26T13:43:33.415 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:3100 IOs, 1340658688 bytes 2023-08-26T13:43:33.504 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:3200 IOs, 1341247488 bytes 2023-08-26T13:43:33.609 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:3300 IOs, 1350148096 bytes 2023-08-26T13:43:33.758 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:3400 IOs, 1380622336 bytes 2023-08-26T13:43:33.929 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:3500 IOs, 1429258240 bytes 2023-08-26T13:43:34.059 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:3600 IOs, 1457404928 bytes 2023-08-26T13:43:34.246 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:3700 IOs, 1510396928 bytes 2023-08-26T13:43:34.403 INFO:tasks.ceph.cluster2.osd.0.smithi124.stderr:2023-08-26T13:43:34.401+0000 7fb60dfa7700 -1 reset not still connected to 0x56452e295380 2023-08-26T13:43:34.417 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:3800 IOs, 1553583104 bytes 2023-08-26T13:43:34.612 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:3900 IOs, 1612771328 bytes 2023-08-26T13:43:34.766 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:4000 IOs, 1638304768 bytes 2023-08-26T13:43:34.877 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:4100 IOs, 1650695168 bytes 2023-08-26T13:43:35.035 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:4200 IOs, 1687323648 bytes 2023-08-26T13:43:35.120 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ error_code=124 2023-08-26T13:43:35.120 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ set -e 2023-08-26T13:43:35.120 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ '[' 124 -eq 124 ']' 2023-08-26T13:43:35.121 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ return 0 2023-08-26T13:43:35.121 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-08-26T13:43:35.121 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ image=image_6 2023-08-26T13:43:35.121 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ create_image_and_enable_mirror cluster2 mirror image_6 snapshot 128M 2023-08-26T13:43:35.121 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster2 2023-08-26T13:43:35.122 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ shift 2023-08-26T13:43:35.122 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T13:43:35.122 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ shift 2023-08-26T13:43:35.122 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=image_6 2023-08-26T13:43:35.122 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ shift 2023-08-26T13:43:35.122 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local mode=snapshot 2023-08-26T13:43:35.123 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ '[' -n snapshot ']' 2023-08-26T13:43:35.123 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ shift 2023-08-26T13:43:35.124 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ create_image cluster2 mirror image_6 128M 2023-08-26T13:43:35.124 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster2 2023-08-26T13:43:35.124 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ shift 2023-08-26T13:43:35.125 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T13:43:35.125 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ shift 2023-08-26T13:43:35.125 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=image_6 2023-08-26T13:43:35.125 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ shift 2023-08-26T13:43:35.125 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local size=128 2023-08-26T13:43:35.126 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ '[' -n 128M ']' 2023-08-26T13:43:35.126 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ size=128M 2023-08-26T13:43:35.126 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ shift 2023-08-26T13:43:35.126 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock mirror/image_6 2023-08-26T13:43:35.177 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ '[' image = image ']' 2023-08-26T13:43:35.178 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ enable_mirror cluster2 mirror image_6 snapshot 2023-08-26T13:43:35.178 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster2 2023-08-26T13:43:35.178 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T13:43:35.178 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=image_6 2023-08-26T13:43:35.179 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local mode=snapshot 2023-08-26T13:43:35.179 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_6 snapshot 2023-08-26T13:43:35.803 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:Mirroring enabled 2023-08-26T13:43:35.809 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rbd --cluster=cluster2 info mirror/image_6 2023-08-26T13:43:35.847 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:rbd image 'image_6': 2023-08-26T13:43:35.847 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: size 128 MiB in 32 objects 2023-08-26T13:43:35.848 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: order 22 (4 MiB objects) 2023-08-26T13:43:35.848 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: snapshot_count: 1 2023-08-26T13:43:35.848 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: id: 12ab75d83ad2 2023-08-26T13:43:35.848 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: block_name_prefix: rbd_data.12ab75d83ad2 2023-08-26T13:43:35.848 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: format: 2 2023-08-26T13:43:35.849 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: features: layering, exclusive-lock 2023-08-26T13:43:35.849 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: op_features: 2023-08-26T13:43:35.849 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: flags: 2023-08-26T13:43:35.849 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: create_timestamp: Sat Aug 26 13:43:35 2023 2023-08-26T13:43:35.849 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: access_timestamp: Sat Aug 26 13:43:35 2023 2023-08-26T13:43:35.850 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: modify_timestamp: Sat Aug 26 13:43:35 2023 2023-08-26T13:43:35.850 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: mirroring state: enabled 2023-08-26T13:43:35.850 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: mirroring mode: snapshot 2023-08-26T13:43:35.850 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: mirroring global id: 42815663-2c57-4ae9-8fb7-0f989c1a91e4 2023-08-26T13:43:35.850 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: mirroring primary: true 2023-08-26T13:43:35.851 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ '[' -n '' ']' 2023-08-26T13:43:35.851 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ stress_write_image cluster2 mirror image_6 2023-08-26T13:43:35.852 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster2 2023-08-26T13:43:35.852 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T13:43:35.852 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=image_6 2023-08-26T13:43:35.852 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2023-08-26T13:43:35.853 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local duration=6 2023-08-26T13:43:35.853 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ set +e 2023-08-26T13:43:35.853 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ timeout 6s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_6 --debug-rbd=20 --debug-journaler=20 2023-08-26T13:43:36.030 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:100 IOs, 29563904 bytes 2023-08-26T13:43:36.118 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:200 IOs, 51555328 bytes 2023-08-26T13:43:36.348 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:300 IOs, 110904320 bytes 2023-08-26T13:43:36.489 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:400 IOs, 140106752 bytes 2023-08-26T13:43:36.634 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:500 IOs, 178729984 bytes 2023-08-26T13:43:36.732 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:600 IOs, 188649472 bytes 2023-08-26T13:43:36.985 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:700 IOs, 279140352 bytes 2023-08-26T13:43:37.141 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:800 IOs, 311756800 bytes 2023-08-26T13:43:37.282 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:900 IOs, 345316352 bytes 2023-08-26T13:43:37.395 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:1000 IOs, 364355584 bytes 2023-08-26T13:43:37.620 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:1100 IOs, 438463488 bytes 2023-08-26T13:43:37.820 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:1200 IOs, 510793728 bytes 2023-08-26T13:43:38.114 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:1300 IOs, 616033280 bytes 2023-08-26T13:43:38.239 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:1400 IOs, 631402496 bytes 2023-08-26T13:43:38.327 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:1500 IOs, 637605888 bytes 2023-08-26T13:43:38.502 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:1600 IOs, 681420800 bytes 2023-08-26T13:43:38.684 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:1700 IOs, 734423040 bytes 2023-08-26T13:43:38.892 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:1800 IOs, 806712320 bytes 2023-08-26T13:43:39.030 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:1900 IOs, 817615872 bytes 2023-08-26T13:43:39.247 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:2000 IOs, 896370688 bytes 2023-08-26T13:43:39.407 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:2100 IOs, 939837440 bytes 2023-08-26T13:43:39.562 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:2200 IOs, 973348864 bytes 2023-08-26T13:43:39.743 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:2300 IOs, 1022915584 bytes 2023-08-26T13:43:39.984 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:2400 IOs, 1108967424 bytes 2023-08-26T13:43:40.219 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:2500 IOs, 1184719872 bytes 2023-08-26T13:43:40.355 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:2600 IOs, 1210324992 bytes 2023-08-26T13:43:40.496 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:2700 IOs, 1234965504 bytes 2023-08-26T13:43:40.645 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:2800 IOs, 1271724032 bytes 2023-08-26T13:43:40.758 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:2900 IOs, 1288707072 bytes 2023-08-26T13:43:40.830 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:3000 IOs, 1289830400 bytes 2023-08-26T13:43:40.998 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:3100 IOs, 1335300096 bytes 2023-08-26T13:43:41.176 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:3200 IOs, 1385031680 bytes 2023-08-26T13:43:41.346 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:3300 IOs, 1452125184 bytes 2023-08-26T13:43:41.542 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:3400 IOs, 1484123136 bytes 2023-08-26T13:43:41.670 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:3500 IOs, 1500379136 bytes 2023-08-26T13:43:41.857 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ error_code=124 2023-08-26T13:43:41.858 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ set -e 2023-08-26T13:43:41.858 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ '[' 124 -eq 124 ']' 2023-08-26T13:43:41.858 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ return 0 2023-08-26T13:43:41.858 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-08-26T13:43:41.858 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ image=image_7 2023-08-26T13:43:41.859 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ create_image_and_enable_mirror cluster2 mirror image_7 snapshot 128M 2023-08-26T13:43:41.859 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster2 2023-08-26T13:43:41.859 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ shift 2023-08-26T13:43:41.859 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T13:43:41.859 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ shift 2023-08-26T13:43:41.860 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=image_7 2023-08-26T13:43:41.860 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ shift 2023-08-26T13:43:41.860 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local mode=snapshot 2023-08-26T13:43:41.860 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ '[' -n snapshot ']' 2023-08-26T13:43:41.860 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ shift 2023-08-26T13:43:41.860 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ create_image cluster2 mirror image_7 128M 2023-08-26T13:43:41.861 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster2 2023-08-26T13:43:41.861 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ shift 2023-08-26T13:43:41.861 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T13:43:41.861 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ shift 2023-08-26T13:43:41.861 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=image_7 2023-08-26T13:43:41.861 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ shift 2023-08-26T13:43:41.862 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local size=128 2023-08-26T13:43:41.862 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ '[' -n 128M ']' 2023-08-26T13:43:41.862 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ size=128M 2023-08-26T13:43:41.862 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ shift 2023-08-26T13:43:41.862 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock mirror/image_7 2023-08-26T13:43:41.900 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ '[' image = image ']' 2023-08-26T13:43:41.900 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ enable_mirror cluster2 mirror image_7 snapshot 2023-08-26T13:43:41.901 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster2 2023-08-26T13:43:41.901 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T13:43:41.901 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=image_7 2023-08-26T13:43:41.901 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local mode=snapshot 2023-08-26T13:43:41.901 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_7 snapshot 2023-08-26T13:43:42.081 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:Mirroring enabled 2023-08-26T13:43:42.088 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rbd --cluster=cluster2 info mirror/image_7 2023-08-26T13:43:42.123 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:rbd image 'image_7': 2023-08-26T13:43:42.124 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: size 128 MiB in 32 objects 2023-08-26T13:43:42.124 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: order 22 (4 MiB objects) 2023-08-26T13:43:42.124 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: snapshot_count: 1 2023-08-26T13:43:42.124 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: id: 12b5f8102cbb 2023-08-26T13:43:42.124 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: block_name_prefix: rbd_data.12b5f8102cbb 2023-08-26T13:43:42.124 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: format: 2 2023-08-26T13:43:42.125 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: features: layering, exclusive-lock 2023-08-26T13:43:42.125 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: op_features: 2023-08-26T13:43:42.125 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: flags: 2023-08-26T13:43:42.125 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: create_timestamp: Sat Aug 26 13:43:41 2023 2023-08-26T13:43:42.125 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: access_timestamp: Sat Aug 26 13:43:41 2023 2023-08-26T13:43:42.125 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: modify_timestamp: Sat Aug 26 13:43:41 2023 2023-08-26T13:43:42.126 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: mirroring state: enabled 2023-08-26T13:43:42.126 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: mirroring mode: snapshot 2023-08-26T13:43:42.126 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: mirroring global id: cdbf4c2c-a074-4ef2-b5df-d98b1449c167 2023-08-26T13:43:42.126 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: mirroring primary: true 2023-08-26T13:43:42.127 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ '[' -n '' ']' 2023-08-26T13:43:42.127 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ stress_write_image cluster2 mirror image_7 2023-08-26T13:43:42.127 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster2 2023-08-26T13:43:42.127 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T13:43:42.128 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=image_7 2023-08-26T13:43:42.128 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2023-08-26T13:43:42.128 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local duration=13 2023-08-26T13:43:42.129 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ set +e 2023-08-26T13:43:42.129 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ timeout 13s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_7 --debug-rbd=20 --debug-journaler=20 2023-08-26T13:43:42.558 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:100 IOs, 129164288 bytes 2023-08-26T13:43:42.826 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:200 IOs, 216744960 bytes 2023-08-26T13:43:42.994 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:300 IOs, 262558720 bytes 2023-08-26T13:43:43.118 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:400 IOs, 277157888 bytes 2023-08-26T13:43:43.325 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:500 IOs, 330430464 bytes 2023-08-26T13:43:43.482 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:600 IOs, 369449984 bytes 2023-08-26T13:43:43.589 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:700 IOs, 384227328 bytes 2023-08-26T13:43:43.684 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:800 IOs, 392303616 bytes 2023-08-26T13:43:43.882 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:900 IOs, 456034304 bytes 2023-08-26T13:43:43.970 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:1000 IOs, 458120192 bytes 2023-08-26T13:43:44.094 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:1100 IOs, 502832128 bytes 2023-08-26T13:43:44.295 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:1200 IOs, 530899968 bytes 2023-08-26T13:43:44.479 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:1300 IOs, 606968832 bytes 2023-08-26T13:43:44.609 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:1400 IOs, 618482688 bytes 2023-08-26T13:43:44.698 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:1500 IOs, 622906368 bytes 2023-08-26T13:43:44.823 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:1600 IOs, 642447360 bytes 2023-08-26T13:43:45.020 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:1700 IOs, 710268928 bytes 2023-08-26T13:43:45.189 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:1800 IOs, 747315200 bytes 2023-08-26T13:43:45.326 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:1900 IOs, 771513344 bytes 2023-08-26T13:43:45.411 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:2000 IOs, 784593920 bytes 2023-08-26T13:43:45.714 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:2100 IOs, 914026496 bytes 2023-08-26T13:43:45.975 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:2200 IOs, 972866560 bytes 2023-08-26T13:43:46.274 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:2300 IOs, 1064498176 bytes 2023-08-26T13:43:46.386 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:2400 IOs, 1074353152 bytes 2023-08-26T13:43:46.553 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:2500 IOs, 1118778368 bytes 2023-08-26T13:43:46.652 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:2600 IOs, 1130246144 bytes 2023-08-26T13:43:46.771 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:2700 IOs, 1146071040 bytes 2023-08-26T13:43:46.886 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:2800 IOs, 1164249088 bytes 2023-08-26T13:43:47.121 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:2900 IOs, 1236337664 bytes 2023-08-26T13:43:47.234 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:3000 IOs, 1255740416 bytes 2023-08-26T13:43:47.333 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:3100 IOs, 1263908864 bytes 2023-08-26T13:43:47.499 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:3200 IOs, 1307053056 bytes 2023-08-26T13:43:47.637 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:3300 IOs, 1353651200 bytes 2023-08-26T13:43:47.808 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:3400 IOs, 1383804928 bytes 2023-08-26T13:43:48.022 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:3500 IOs, 1448473600 bytes 2023-08-26T13:43:48.218 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:3600 IOs, 1496827904 bytes 2023-08-26T13:43:48.399 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:3700 IOs, 1546747904 bytes 2023-08-26T13:43:48.492 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:3800 IOs, 1547373568 bytes 2023-08-26T13:43:48.695 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:3900 IOs, 1603255296 bytes 2023-08-26T13:43:48.864 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:4000 IOs, 1641835520 bytes 2023-08-26T13:43:48.991 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:4100 IOs, 1668239360 bytes 2023-08-26T13:43:49.088 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:4200 IOs, 1683557376 bytes 2023-08-26T13:43:49.286 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:4300 IOs, 1727861760 bytes 2023-08-26T13:43:49.426 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:4400 IOs, 1752840192 bytes 2023-08-26T13:43:49.551 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:4500 IOs, 1769368576 bytes 2023-08-26T13:43:49.751 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:4600 IOs, 1827459072 bytes 2023-08-26T13:43:49.840 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:4700 IOs, 1828056064 bytes 2023-08-26T13:43:50.005 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:4800 IOs, 1874052096 bytes 2023-08-26T13:43:50.161 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:4900 IOs, 1908677632 bytes 2023-08-26T13:43:50.312 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:5000 IOs, 1939321856 bytes 2023-08-26T13:43:50.475 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:5100 IOs, 1982752768 bytes 2023-08-26T13:43:50.628 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:5200 IOs, 2018120704 bytes 2023-08-26T13:43:50.825 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:5300 IOs, 2065894400 bytes 2023-08-26T13:43:50.960 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:5400 IOs, 2095380480 bytes 2023-08-26T13:43:51.159 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:5500 IOs, 2148984832 bytes 2023-08-26T13:43:51.347 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:5600 IOs, 2202775552 bytes 2023-08-26T13:43:51.609 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:5700 IOs, 2294966272 bytes 2023-08-26T13:43:51.824 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:5800 IOs, 2368358400 bytes 2023-08-26T13:43:51.988 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:5900 IOs, 2379019264 bytes 2023-08-26T13:43:52.127 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:6000 IOs, 2415926272 bytes 2023-08-26T13:43:52.401 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:6100 IOs, 2504337408 bytes 2023-08-26T13:43:52.497 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:6200 IOs, 2514277376 bytes 2023-08-26T13:43:52.596 INFO:tasks.rbd_mirror_thrash:kill cluster1.client.mirror.0 2023-08-26T13:43:52.597 INFO:tasks.rbd_mirror.cluster1.client.mirror.0:Sent signal 15 2023-08-26T13:43:52.597 INFO:tasks.rbd_mirror_thrash:waiting for 2 secs before reviving daemons 2023-08-26T13:43:52.598 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi124.stderr:2023-08-26T13:43:52.597+0000 7fe8ae399700 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.0 (PID: 114265) UID: 1000 2023-08-26T13:43:52.647 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:6300 IOs, 2546826240 bytes 2023-08-26T13:43:52.758 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:6400 IOs, 2551400448 bytes 2023-08-26T13:43:52.937 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:6500 IOs, 2613607424 bytes 2023-08-26T13:43:53.130 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:6600 IOs, 2643021824 bytes 2023-08-26T13:43:53.261 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:6700 IOs, 2669234176 bytes 2023-08-26T13:43:53.392 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:6800 IOs, 2693601280 bytes 2023-08-26T13:43:53.517 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:6900 IOs, 2705026048 bytes 2023-08-26T13:43:53.700 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:7000 IOs, 2739197952 bytes 2023-08-26T13:43:53.936 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:7100 IOs, 2761026560 bytes 2023-08-26T13:43:54.224 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:7200 IOs, 2855574528 bytes 2023-08-26T13:43:54.381 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:7300 IOs, 2906037248 bytes 2023-08-26T13:43:54.496 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:7400 IOs, 2908242944 bytes 2023-08-26T13:43:54.598 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:7500 IOs, 2918391808 bytes 2023-08-26T13:43:54.598 INFO:tasks.rbd_mirror_thrash:waiting for cluster1.client.mirror.0 2023-08-26T13:43:54.598 INFO:teuthology.orchestra.run:waiting for 600 2023-08-26T13:43:54.599 INFO:tasks.rbd_mirror_thrash:reviving cluster1.client.mirror.0 2023-08-26T13:43:54.599 INFO:tasks.rbd_mirror.cluster1.client.mirror.0:Restarting daemon 2023-08-26T13:43:54.599 DEBUG:teuthology.orchestra.run.smithi124:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.0 2023-08-26T13:43:54.602 INFO:tasks.rbd_mirror.cluster1.client.mirror.0:Started 2023-08-26T13:43:54.602 INFO:tasks.rbd_mirror_thrash:waiting for 117 secs before thrashing 2023-08-26T13:43:54.815 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:7600 IOs, 2963860480 bytes 2023-08-26T13:43:54.986 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:7700 IOs, 3013143552 bytes 2023-08-26T13:43:55.134 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ error_code=124 2023-08-26T13:43:55.134 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ set -e 2023-08-26T13:43:55.135 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ '[' 124 -eq 124 ']' 2023-08-26T13:43:55.135 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ return 0 2023-08-26T13:43:55.135 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-08-26T13:43:55.135 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ image=image_8 2023-08-26T13:43:55.135 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ create_image_and_enable_mirror cluster2 mirror image_8 snapshot 128M 2023-08-26T13:43:55.136 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster2 2023-08-26T13:43:55.136 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ shift 2023-08-26T13:43:55.136 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T13:43:55.136 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ shift 2023-08-26T13:43:55.136 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=image_8 2023-08-26T13:43:55.136 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ shift 2023-08-26T13:43:55.137 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local mode=snapshot 2023-08-26T13:43:55.137 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ '[' -n snapshot ']' 2023-08-26T13:43:55.137 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ shift 2023-08-26T13:43:55.137 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ create_image cluster2 mirror image_8 128M 2023-08-26T13:43:55.137 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster2 2023-08-26T13:43:55.138 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ shift 2023-08-26T13:43:55.138 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T13:43:55.138 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ shift 2023-08-26T13:43:55.138 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=image_8 2023-08-26T13:43:55.138 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ shift 2023-08-26T13:43:55.139 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local size=128 2023-08-26T13:43:55.139 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ '[' -n 128M ']' 2023-08-26T13:43:55.139 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ size=128M 2023-08-26T13:43:55.139 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ shift 2023-08-26T13:43:55.139 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock mirror/image_8 2023-08-26T13:43:55.177 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ '[' image = image ']' 2023-08-26T13:43:55.178 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ enable_mirror cluster2 mirror image_8 snapshot 2023-08-26T13:43:55.178 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster2 2023-08-26T13:43:55.178 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T13:43:55.179 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=image_8 2023-08-26T13:43:55.179 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local mode=snapshot 2023-08-26T13:43:55.179 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_8 snapshot 2023-08-26T13:43:55.831 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:Mirroring enabled 2023-08-26T13:43:55.838 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rbd --cluster=cluster2 info mirror/image_8 2023-08-26T13:43:55.876 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:rbd image 'image_8': 2023-08-26T13:43:55.876 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: size 128 MiB in 32 objects 2023-08-26T13:43:55.876 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: order 22 (4 MiB objects) 2023-08-26T13:43:55.876 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: snapshot_count: 1 2023-08-26T13:43:55.877 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: id: 12c3d182eba4 2023-08-26T13:43:55.877 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: block_name_prefix: rbd_data.12c3d182eba4 2023-08-26T13:43:55.877 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: format: 2 2023-08-26T13:43:55.877 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: features: layering, exclusive-lock 2023-08-26T13:43:55.878 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: op_features: 2023-08-26T13:43:55.878 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: flags: 2023-08-26T13:43:55.878 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: create_timestamp: Sat Aug 26 13:43:55 2023 2023-08-26T13:43:55.878 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: access_timestamp: Sat Aug 26 13:43:55 2023 2023-08-26T13:43:55.878 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: modify_timestamp: Sat Aug 26 13:43:55 2023 2023-08-26T13:43:55.879 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: mirroring state: enabled 2023-08-26T13:43:55.879 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: mirroring mode: snapshot 2023-08-26T13:43:55.879 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: mirroring global id: 0e8ffb4e-bbd1-41aa-b41a-6b2aebda2762 2023-08-26T13:43:55.879 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: mirroring primary: true 2023-08-26T13:43:55.880 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ '[' -n '' ']' 2023-08-26T13:43:55.880 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ stress_write_image cluster2 mirror image_8 2023-08-26T13:43:55.881 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster2 2023-08-26T13:43:55.881 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T13:43:55.881 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=image_8 2023-08-26T13:43:55.881 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2023-08-26T13:43:55.882 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local duration=7 2023-08-26T13:43:55.882 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ set +e 2023-08-26T13:43:55.882 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ timeout 7s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_8 --debug-rbd=20 --debug-journaler=20 2023-08-26T13:43:56.191 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:100 IOs, 88567808 bytes 2023-08-26T13:43:56.417 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:200 IOs, 147734528 bytes 2023-08-26T13:43:56.553 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:300 IOs, 168691712 bytes 2023-08-26T13:43:56.812 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:400 IOs, 244783104 bytes 2023-08-26T13:43:57.008 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:500 IOs, 295599104 bytes 2023-08-26T13:43:57.137 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:600 IOs, 306951168 bytes 2023-08-26T13:43:57.234 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:700 IOs, 316114944 bytes 2023-08-26T13:43:57.368 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:800 IOs, 329142272 bytes 2023-08-26T13:43:57.558 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:900 IOs, 372733952 bytes 2023-08-26T13:43:57.721 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:1000 IOs, 406750208 bytes 2023-08-26T13:43:57.896 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:1100 IOs, 447166464 bytes 2023-08-26T13:43:58.051 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:1200 IOs, 472472576 bytes 2023-08-26T13:43:58.180 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:1300 IOs, 488769536 bytes 2023-08-26T13:43:58.366 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:1400 IOs, 546161664 bytes 2023-08-26T13:43:58.664 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:1500 IOs, 643548160 bytes 2023-08-26T13:43:58.925 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:1600 IOs, 708998144 bytes 2023-08-26T13:43:59.067 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:1700 IOs, 726213632 bytes 2023-08-26T13:43:59.253 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:1800 IOs, 785111040 bytes 2023-08-26T13:43:59.487 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:1900 IOs, 840285184 bytes 2023-08-26T13:43:59.626 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:2000 IOs, 863815680 bytes 2023-08-26T13:43:59.747 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:2100 IOs, 882091008 bytes 2023-08-26T13:43:59.971 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:2200 IOs, 931589120 bytes 2023-08-26T13:44:00.316 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:2300 IOs, 1049839616 bytes 2023-08-26T13:44:00.528 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:2400 IOs, 1108632576 bytes 2023-08-26T13:44:00.663 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:2500 IOs, 1132680192 bytes 2023-08-26T13:44:00.807 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:2600 IOs, 1155074048 bytes 2023-08-26T13:44:00.941 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:2700 IOs, 1179325440 bytes 2023-08-26T13:44:01.036 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:2800 IOs, 1179949056 bytes 2023-08-26T13:44:01.178 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:2900 IOs, 1207789568 bytes 2023-08-26T13:44:01.272 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:3000 IOs, 1208415232 bytes 2023-08-26T13:44:01.496 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:3100 IOs, 1273098240 bytes 2023-08-26T13:44:01.747 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:3200 IOs, 1343921152 bytes 2023-08-26T13:44:01.910 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:3300 IOs, 1393461248 bytes 2023-08-26T13:44:02.085 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:3400 IOs, 1409507328 bytes 2023-08-26T13:44:02.301 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:3500 IOs, 1494252544 bytes 2023-08-26T13:44:02.505 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:3600 IOs, 1514481664 bytes 2023-08-26T13:44:02.676 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:3700 IOs, 1549283328 bytes 2023-08-26T13:44:02.863 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:3800 IOs, 1598101504 bytes 2023-08-26T13:44:02.887 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ error_code=124 2023-08-26T13:44:02.888 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ set -e 2023-08-26T13:44:02.888 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ '[' 124 -eq 124 ']' 2023-08-26T13:44:02.888 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ return 0 2023-08-26T13:44:02.888 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-08-26T13:44:02.889 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ image=image_9 2023-08-26T13:44:02.889 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ create_image_and_enable_mirror cluster2 mirror image_9 snapshot 128M 2023-08-26T13:44:02.889 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster2 2023-08-26T13:44:02.889 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ shift 2023-08-26T13:44:02.889 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T13:44:02.890 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ shift 2023-08-26T13:44:02.890 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=image_9 2023-08-26T13:44:02.890 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ shift 2023-08-26T13:44:02.890 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local mode=snapshot 2023-08-26T13:44:02.891 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ '[' -n snapshot ']' 2023-08-26T13:44:02.891 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ shift 2023-08-26T13:44:02.891 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ create_image cluster2 mirror image_9 128M 2023-08-26T13:44:02.891 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster2 2023-08-26T13:44:02.892 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ shift 2023-08-26T13:44:02.892 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T13:44:02.892 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ shift 2023-08-26T13:44:02.892 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=image_9 2023-08-26T13:44:02.892 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ shift 2023-08-26T13:44:02.893 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local size=128 2023-08-26T13:44:02.893 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ '[' -n 128M ']' 2023-08-26T13:44:02.893 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ size=128M 2023-08-26T13:44:02.893 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ shift 2023-08-26T13:44:02.893 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock mirror/image_9 2023-08-26T13:44:02.932 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ '[' image = image ']' 2023-08-26T13:44:02.933 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ enable_mirror cluster2 mirror image_9 snapshot 2023-08-26T13:44:02.933 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster2 2023-08-26T13:44:02.933 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T13:44:02.933 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=image_9 2023-08-26T13:44:02.934 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local mode=snapshot 2023-08-26T13:44:02.934 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_9 snapshot 2023-08-26T13:44:03.081 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:Mirroring enabled 2023-08-26T13:44:03.088 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rbd --cluster=cluster2 info mirror/image_9 2023-08-26T13:44:03.123 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:rbd image 'image_9': 2023-08-26T13:44:03.124 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: size 128 MiB in 32 objects 2023-08-26T13:44:03.124 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: order 22 (4 MiB objects) 2023-08-26T13:44:03.124 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: snapshot_count: 1 2023-08-26T13:44:03.124 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: id: 12cb2fb6e768 2023-08-26T13:44:03.124 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: block_name_prefix: rbd_data.12cb2fb6e768 2023-08-26T13:44:03.125 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: format: 2 2023-08-26T13:44:03.125 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: features: layering, exclusive-lock 2023-08-26T13:44:03.125 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: op_features: 2023-08-26T13:44:03.125 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: flags: 2023-08-26T13:44:03.126 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: create_timestamp: Sat Aug 26 13:44:02 2023 2023-08-26T13:44:03.126 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: access_timestamp: Sat Aug 26 13:44:02 2023 2023-08-26T13:44:03.126 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: modify_timestamp: Sat Aug 26 13:44:02 2023 2023-08-26T13:44:03.126 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: mirroring state: enabled 2023-08-26T13:44:03.126 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: mirroring mode: snapshot 2023-08-26T13:44:03.127 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: mirroring global id: 680b775a-fecc-478c-ab5f-9dc6f2f67724 2023-08-26T13:44:03.127 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: mirroring primary: true 2023-08-26T13:44:03.128 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ '[' -n '' ']' 2023-08-26T13:44:03.128 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ stress_write_image cluster2 mirror image_9 2023-08-26T13:44:03.128 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster2 2023-08-26T13:44:03.129 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T13:44:03.129 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=image_9 2023-08-26T13:44:03.129 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2023-08-26T13:44:03.129 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local duration=10 2023-08-26T13:44:03.129 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ set +e 2023-08-26T13:44:03.130 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ timeout 10s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_9 --debug-rbd=20 --debug-journaler=20 2023-08-26T13:44:03.344 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:100 IOs, 41011200 bytes 2023-08-26T13:44:03.481 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:200 IOs, 66335744 bytes 2023-08-26T13:44:03.617 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:300 IOs, 94147584 bytes 2023-08-26T13:44:03.764 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:400 IOs, 117607424 bytes 2023-08-26T13:44:03.946 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:500 IOs, 156910592 bytes 2023-08-26T13:44:04.081 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:600 IOs, 179838976 bytes 2023-08-26T13:44:04.211 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:700 IOs, 210479104 bytes 2023-08-26T13:44:04.304 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:800 IOs, 213323776 bytes 2023-08-26T13:44:04.507 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:900 IOs, 256404480 bytes 2023-08-26T13:44:04.755 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:1000 IOs, 340545536 bytes 2023-08-26T13:44:04.942 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:1100 IOs, 364068864 bytes 2023-08-26T13:44:05.048 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:1200 IOs, 373619712 bytes 2023-08-26T13:44:05.176 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:1300 IOs, 391793664 bytes 2023-08-26T13:44:05.409 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:1400 IOs, 463414272 bytes 2023-08-26T13:44:05.552 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:1500 IOs, 478871552 bytes 2023-08-26T13:44:05.689 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:1600 IOs, 502352896 bytes 2023-08-26T13:44:05.872 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:1700 IOs, 547419136 bytes 2023-08-26T13:44:06.038 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:1800 IOs, 587753472 bytes 2023-08-26T13:44:06.179 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:1900 IOs, 606326784 bytes 2023-08-26T13:44:06.301 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:2000 IOs, 625520640 bytes 2023-08-26T13:44:06.488 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:2100 IOs, 677470208 bytes 2023-08-26T13:44:06.727 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:2200 IOs, 725755904 bytes 2023-08-26T13:44:06.909 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:2300 IOs, 791585792 bytes 2023-08-26T13:44:07.119 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:2400 IOs, 832659456 bytes 2023-08-26T13:44:07.279 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:2500 IOs, 868567040 bytes 2023-08-26T13:44:07.394 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:2600 IOs, 891173888 bytes 2023-08-26T13:44:07.716 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:2700 IOs, 985497600 bytes 2023-08-26T13:44:07.910 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:2800 IOs, 1030339584 bytes 2023-08-26T13:44:08.172 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:2900 IOs, 1118928896 bytes 2023-08-26T13:44:08.346 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:3000 IOs, 1139628032 bytes 2023-08-26T13:44:08.464 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:3100 IOs, 1146861568 bytes 2023-08-26T13:44:08.625 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:3200 IOs, 1179716608 bytes 2023-08-26T13:44:08.787 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:3300 IOs, 1214703616 bytes 2023-08-26T13:44:08.921 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:3400 IOs, 1233850368 bytes 2023-08-26T13:44:09.086 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:3500 IOs, 1271743488 bytes 2023-08-26T13:44:09.247 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:3600 IOs, 1305453568 bytes 2023-08-26T13:44:09.403 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:3700 IOs, 1335590912 bytes 2023-08-26T13:44:09.538 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:3800 IOs, 1357416448 bytes 2023-08-26T13:44:09.763 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:3900 IOs, 1431206912 bytes 2023-08-26T13:44:09.919 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:4000 IOs, 1468569600 bytes 2023-08-26T13:44:10.159 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:4100 IOs, 1531752448 bytes 2023-08-26T13:44:10.258 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:4200 IOs, 1532340224 bytes 2023-08-26T13:44:10.435 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:4300 IOs, 1580797952 bytes 2023-08-26T13:44:10.591 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:4400 IOs, 1617631232 bytes 2023-08-26T13:44:10.727 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:4500 IOs, 1643326464 bytes 2023-08-26T13:44:10.807 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:4600 IOs, 1658953728 bytes 2023-08-26T13:44:11.007 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:4700 IOs, 1703098368 bytes 2023-08-26T13:44:11.264 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:4800 IOs, 1795456000 bytes 2023-08-26T13:44:11.434 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:4900 IOs, 1809115136 bytes 2023-08-26T13:44:11.671 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:5000 IOs, 1874365440 bytes 2023-08-26T13:44:11.849 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:5100 IOs, 1922107392 bytes 2023-08-26T13:44:12.001 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:5200 IOs, 1955126272 bytes 2023-08-26T13:44:12.234 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:5300 IOs, 2033516544 bytes 2023-08-26T13:44:12.343 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:5400 IOs, 2034135040 bytes 2023-08-26T13:44:12.490 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:5500 IOs, 2061277184 bytes 2023-08-26T13:44:12.754 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:5600 IOs, 2159381504 bytes 2023-08-26T13:44:12.962 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:5700 IOs, 2212769792 bytes 2023-08-26T13:44:13.134 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ error_code=124 2023-08-26T13:44:13.134 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ set -e 2023-08-26T13:44:13.134 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ '[' 124 -eq 124 ']' 2023-08-26T13:44:13.135 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ return 0 2023-08-26T13:44:13.135 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-08-26T13:44:13.135 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ image=image_10 2023-08-26T13:44:13.135 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ create_image_and_enable_mirror cluster2 mirror image_10 snapshot 128M 2023-08-26T13:44:13.135 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster2 2023-08-26T13:44:13.136 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ shift 2023-08-26T13:44:13.136 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T13:44:13.136 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ shift 2023-08-26T13:44:13.136 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=image_10 2023-08-26T13:44:13.136 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ shift 2023-08-26T13:44:13.137 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local mode=snapshot 2023-08-26T13:44:13.137 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ '[' -n snapshot ']' 2023-08-26T13:44:13.137 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ shift 2023-08-26T13:44:13.137 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ create_image cluster2 mirror image_10 128M 2023-08-26T13:44:13.137 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster2 2023-08-26T13:44:13.138 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ shift 2023-08-26T13:44:13.138 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T13:44:13.138 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ shift 2023-08-26T13:44:13.138 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=image_10 2023-08-26T13:44:13.138 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ shift 2023-08-26T13:44:13.139 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local size=128 2023-08-26T13:44:13.139 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ '[' -n 128M ']' 2023-08-26T13:44:13.139 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ size=128M 2023-08-26T13:44:13.139 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ shift 2023-08-26T13:44:13.139 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock mirror/image_10 2023-08-26T13:44:13.178 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ '[' image = image ']' 2023-08-26T13:44:13.178 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ enable_mirror cluster2 mirror image_10 snapshot 2023-08-26T13:44:13.178 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster2 2023-08-26T13:44:13.178 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T13:44:13.179 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=image_10 2023-08-26T13:44:13.179 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local mode=snapshot 2023-08-26T13:44:13.179 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_10 snapshot 2023-08-26T13:44:14.182 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:Mirroring enabled 2023-08-26T13:44:14.190 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rbd --cluster=cluster2 info mirror/image_10 2023-08-26T13:44:14.228 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:rbd image 'image_10': 2023-08-26T13:44:14.229 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: size 128 MiB in 32 objects 2023-08-26T13:44:14.229 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: order 22 (4 MiB objects) 2023-08-26T13:44:14.229 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: snapshot_count: 1 2023-08-26T13:44:14.229 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: id: 12d4448e9644 2023-08-26T13:44:14.229 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: block_name_prefix: rbd_data.12d4448e9644 2023-08-26T13:44:14.230 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: format: 2 2023-08-26T13:44:14.230 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: features: layering, exclusive-lock 2023-08-26T13:44:14.230 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: op_features: 2023-08-26T13:44:14.230 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: flags: 2023-08-26T13:44:14.230 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: create_timestamp: Sat Aug 26 13:44:13 2023 2023-08-26T13:44:14.231 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: access_timestamp: Sat Aug 26 13:44:13 2023 2023-08-26T13:44:14.231 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: modify_timestamp: Sat Aug 26 13:44:13 2023 2023-08-26T13:44:14.231 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: mirroring state: enabled 2023-08-26T13:44:14.231 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: mirroring mode: snapshot 2023-08-26T13:44:14.231 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: mirroring global id: 83466149-7311-4c4c-898e-a67f6a0dbe7c 2023-08-26T13:44:14.231 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: mirroring primary: true 2023-08-26T13:44:14.232 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ '[' -n '' ']' 2023-08-26T13:44:14.232 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ stress_write_image cluster2 mirror image_10 2023-08-26T13:44:14.233 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster2 2023-08-26T13:44:14.233 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T13:44:14.233 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=image_10 2023-08-26T13:44:14.233 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2023-08-26T13:44:14.234 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local duration=7 2023-08-26T13:44:14.234 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ set +e 2023-08-26T13:44:14.234 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ timeout 7s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_10 --debug-rbd=20 --debug-journaler=20 2023-08-26T13:44:14.425 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:100 IOs, 25300992 bytes 2023-08-26T13:44:14.491 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.0 with skip (0.9015863758370269) > weight (0.25) 2023-08-26T13:44:14.492 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.1 with skip (0.25751621725256835) > weight (0.25) 2023-08-26T13:44:14.492 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.2 with skip (0.39207829207739675) > weight (0.25) 2023-08-26T13:44:14.492 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.3 with skip (0.6082731566496503) > weight (0.25) 2023-08-26T13:44:14.492 INFO:tasks.rbd_mirror_thrash:waiting for 116 secs before thrashing 2023-08-26T13:44:14.613 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:200 IOs, 75236352 bytes 2023-08-26T13:44:14.816 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:300 IOs, 151778304 bytes 2023-08-26T13:44:15.026 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:400 IOs, 191929344 bytes 2023-08-26T13:44:15.124 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:500 IOs, 199005184 bytes 2023-08-26T13:44:15.283 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:600 IOs, 235355136 bytes 2023-08-26T13:44:15.423 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:700 IOs, 250591232 bytes 2023-08-26T13:44:15.669 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:800 IOs, 316853248 bytes 2023-08-26T13:44:15.834 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:900 IOs, 356874240 bytes 2023-08-26T13:44:15.980 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:1000 IOs, 384945152 bytes 2023-08-26T13:44:16.169 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:1100 IOs, 461313024 bytes 2023-08-26T13:44:16.386 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:1200 IOs, 494606336 bytes 2023-08-26T13:44:16.585 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:1300 IOs, 552947712 bytes 2023-08-26T13:44:16.777 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:1400 IOs, 612472832 bytes 2023-08-26T13:44:16.895 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:1500 IOs, 626481152 bytes 2023-08-26T13:44:17.052 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:1600 IOs, 668754944 bytes 2023-08-26T13:44:17.325 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:1700 IOs, 756930560 bytes 2023-08-26T13:44:17.506 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:1800 IOs, 807128064 bytes 2023-08-26T13:44:17.673 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:1900 IOs, 840359936 bytes 2023-08-26T13:44:17.775 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:2000 IOs, 846156800 bytes 2023-08-26T13:44:17.963 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:2100 IOs, 891197440 bytes 2023-08-26T13:44:18.144 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:2200 IOs, 932826112 bytes 2023-08-26T13:44:18.355 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:2300 IOs, 992937984 bytes 2023-08-26T13:44:18.479 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:2400 IOs, 1006013440 bytes 2023-08-26T13:44:18.602 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:2500 IOs, 1024313344 bytes 2023-08-26T13:44:18.830 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:2600 IOs, 1102276608 bytes 2023-08-26T13:44:19.015 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:2700 IOs, 1148512256 bytes 2023-08-26T13:44:19.159 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:2800 IOs, 1180087296 bytes 2023-08-26T13:44:19.293 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:2900 IOs, 1209209856 bytes 2023-08-26T13:44:19.458 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:3000 IOs, 1242841088 bytes 2023-08-26T13:44:19.630 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:3100 IOs, 1290769408 bytes 2023-08-26T13:44:19.810 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:3200 IOs, 1336441856 bytes 2023-08-26T13:44:20.011 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:3300 IOs, 1396585472 bytes 2023-08-26T13:44:20.123 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:3400 IOs, 1408105472 bytes 2023-08-26T13:44:20.252 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:3500 IOs, 1424902144 bytes 2023-08-26T13:44:20.535 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:3600 IOs, 1521608704 bytes 2023-08-26T13:44:20.629 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:3700 IOs, 1522227200 bytes 2023-08-26T13:44:20.737 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:3800 IOs, 1533136896 bytes 2023-08-26T13:44:20.870 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:3900 IOs, 1550582784 bytes 2023-08-26T13:44:21.016 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:4000 IOs, 1577064448 bytes 2023-08-26T13:44:21.240 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ error_code=124 2023-08-26T13:44:21.240 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ set -e 2023-08-26T13:44:21.240 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ '[' 124 -eq 124 ']' 2023-08-26T13:44:21.240 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ return 0 2023-08-26T13:44:21.241 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-08-26T13:44:21.241 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ image=image_11 2023-08-26T13:44:21.241 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ create_image_and_enable_mirror cluster2 mirror image_11 snapshot 128M 2023-08-26T13:44:21.241 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster2 2023-08-26T13:44:21.241 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ shift 2023-08-26T13:44:21.242 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T13:44:21.242 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ shift 2023-08-26T13:44:21.242 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=image_11 2023-08-26T13:44:21.242 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ shift 2023-08-26T13:44:21.242 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local mode=snapshot 2023-08-26T13:44:21.242 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ '[' -n snapshot ']' 2023-08-26T13:44:21.243 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ shift 2023-08-26T13:44:21.243 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ create_image cluster2 mirror image_11 128M 2023-08-26T13:44:21.243 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster2 2023-08-26T13:44:21.243 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ shift 2023-08-26T13:44:21.243 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T13:44:21.244 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ shift 2023-08-26T13:44:21.244 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=image_11 2023-08-26T13:44:21.244 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ shift 2023-08-26T13:44:21.244 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local size=128 2023-08-26T13:44:21.244 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ '[' -n 128M ']' 2023-08-26T13:44:21.245 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ size=128M 2023-08-26T13:44:21.245 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ shift 2023-08-26T13:44:21.245 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock mirror/image_11 2023-08-26T13:44:21.285 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ '[' image = image ']' 2023-08-26T13:44:21.285 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ enable_mirror cluster2 mirror image_11 snapshot 2023-08-26T13:44:21.285 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster2 2023-08-26T13:44:21.286 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T13:44:21.286 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=image_11 2023-08-26T13:44:21.286 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local mode=snapshot 2023-08-26T13:44:21.286 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_11 snapshot 2023-08-26T13:44:22.228 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:Mirroring enabled 2023-08-26T13:44:22.236 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rbd --cluster=cluster2 info mirror/image_11 2023-08-26T13:44:22.276 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:rbd image 'image_11': 2023-08-26T13:44:22.276 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: size 128 MiB in 32 objects 2023-08-26T13:44:22.276 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: order 22 (4 MiB objects) 2023-08-26T13:44:22.276 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: snapshot_count: 1 2023-08-26T13:44:22.276 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: id: 12dd9a3cdba4 2023-08-26T13:44:22.277 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: block_name_prefix: rbd_data.12dd9a3cdba4 2023-08-26T13:44:22.277 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: format: 2 2023-08-26T13:44:22.277 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: features: layering, exclusive-lock 2023-08-26T13:44:22.277 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: op_features: 2023-08-26T13:44:22.277 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: flags: 2023-08-26T13:44:22.278 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: create_timestamp: Sat Aug 26 13:44:21 2023 2023-08-26T13:44:22.278 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: access_timestamp: Sat Aug 26 13:44:21 2023 2023-08-26T13:44:22.278 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: modify_timestamp: Sat Aug 26 13:44:21 2023 2023-08-26T13:44:22.278 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: mirroring state: enabled 2023-08-26T13:44:22.279 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: mirroring mode: snapshot 2023-08-26T13:44:22.279 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: mirroring global id: f979d3c9-8b28-4180-872b-79e338751061 2023-08-26T13:44:22.279 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: mirroring primary: true 2023-08-26T13:44:22.280 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ '[' -n '' ']' 2023-08-26T13:44:22.280 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ stress_write_image cluster2 mirror image_11 2023-08-26T13:44:22.280 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster2 2023-08-26T13:44:22.280 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T13:44:22.281 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=image_11 2023-08-26T13:44:22.281 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2023-08-26T13:44:22.281 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local duration=9 2023-08-26T13:44:22.282 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ set +e 2023-08-26T13:44:22.282 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ timeout 9s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_11 --debug-rbd=20 --debug-journaler=20 2023-08-26T13:44:22.578 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:100 IOs, 86419456 bytes 2023-08-26T13:44:22.731 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:200 IOs, 106875904 bytes 2023-08-26T13:44:22.859 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:300 IOs, 135542784 bytes 2023-08-26T13:44:22.970 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:400 IOs, 140861440 bytes 2023-08-26T13:44:23.279 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:500 IOs, 250587136 bytes 2023-08-26T13:44:23.482 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:600 IOs, 316998656 bytes 2023-08-26T13:44:23.797 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:700 IOs, 408690688 bytes 2023-08-26T13:44:24.097 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:800 IOs, 477527040 bytes 2023-08-26T13:44:24.319 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:900 IOs, 551851008 bytes 2023-08-26T13:44:24.499 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:1000 IOs, 583130112 bytes 2023-08-26T13:44:24.734 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:1100 IOs, 626726912 bytes 2023-08-26T13:44:24.877 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:1200 IOs, 663319552 bytes 2023-08-26T13:44:25.031 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:1300 IOs, 696124416 bytes 2023-08-26T13:44:25.144 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:1400 IOs, 719797248 bytes 2023-08-26T13:44:25.439 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:1500 IOs, 811510784 bytes 2023-08-26T13:44:25.681 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:1600 IOs, 889645056 bytes 2023-08-26T13:44:25.827 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:1700 IOs, 919335936 bytes 2023-08-26T13:44:25.925 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:1800 IOs, 934080512 bytes 2023-08-26T13:44:26.113 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:1900 IOs, 976147456 bytes 2023-08-26T13:44:26.248 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:2000 IOs, 998476800 bytes 2023-08-26T13:44:26.407 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:2100 IOs, 1033329664 bytes 2023-08-26T13:44:26.530 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:2200 IOs, 1057216512 bytes 2023-08-26T13:44:26.672 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:2300 IOs, 1079064576 bytes 2023-08-26T13:44:26.976 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:2400 IOs, 1183649792 bytes 2023-08-26T13:44:27.168 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:2500 IOs, 1235428352 bytes 2023-08-26T13:44:27.283 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:2600 IOs, 1254645760 bytes 2023-08-26T13:44:27.425 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:2700 IOs, 1285392384 bytes 2023-08-26T13:44:27.549 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:2800 IOs, 1303535616 bytes 2023-08-26T13:44:27.747 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:2900 IOs, 1360343040 bytes 2023-08-26T13:44:27.974 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:3000 IOs, 1434156032 bytes 2023-08-26T13:44:28.142 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:3100 IOs, 1464692736 bytes 2023-08-26T13:44:28.321 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:3200 IOs, 1504516096 bytes 2023-08-26T13:44:28.540 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:3300 IOs, 1565450240 bytes 2023-08-26T13:44:28.670 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:3400 IOs, 1590441984 bytes 2023-08-26T13:44:28.904 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:3500 IOs, 1681090560 bytes 2023-08-26T13:44:29.111 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:3600 IOs, 1722605568 bytes 2023-08-26T13:44:29.253 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:3700 IOs, 1743328256 bytes 2023-08-26T13:44:29.367 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:3800 IOs, 1771855872 bytes 2023-08-26T13:44:29.525 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:3900 IOs, 1793417216 bytes 2023-08-26T13:44:29.684 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:4000 IOs, 1834482688 bytes 2023-08-26T13:44:29.781 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:4100 IOs, 1838188544 bytes 2023-08-26T13:44:29.884 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:4200 IOs, 1851983872 bytes 2023-08-26T13:44:30.055 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:4300 IOs, 1891446784 bytes 2023-08-26T13:44:30.180 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:4400 IOs, 1914443776 bytes 2023-08-26T13:44:30.352 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:4500 IOs, 1955480576 bytes 2023-08-26T13:44:30.513 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:4600 IOs, 2011658240 bytes 2023-08-26T13:44:30.728 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:4700 IOs, 2052814848 bytes 2023-08-26T13:44:30.842 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:4800 IOs, 2063215616 bytes 2023-08-26T13:44:31.060 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:4900 IOs, 2135098368 bytes 2023-08-26T13:44:31.155 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:5000 IOs, 2143407104 bytes 2023-08-26T13:44:31.290 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ error_code=124 2023-08-26T13:44:31.291 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ set -e 2023-08-26T13:44:31.291 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ '[' 124 -eq 124 ']' 2023-08-26T13:44:31.291 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ return 0 2023-08-26T13:44:31.291 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-08-26T13:44:31.291 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ image=image_12 2023-08-26T13:44:31.292 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ create_image_and_enable_mirror cluster2 mirror image_12 snapshot 128M 2023-08-26T13:44:31.292 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster2 2023-08-26T13:44:31.292 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ shift 2023-08-26T13:44:31.292 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T13:44:31.292 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ shift 2023-08-26T13:44:31.293 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=image_12 2023-08-26T13:44:31.293 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ shift 2023-08-26T13:44:31.293 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local mode=snapshot 2023-08-26T13:44:31.293 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ '[' -n snapshot ']' 2023-08-26T13:44:31.293 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ shift 2023-08-26T13:44:31.293 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ create_image cluster2 mirror image_12 128M 2023-08-26T13:44:31.294 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster2 2023-08-26T13:44:31.294 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ shift 2023-08-26T13:44:31.294 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T13:44:31.294 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ shift 2023-08-26T13:44:31.294 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=image_12 2023-08-26T13:44:31.295 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ shift 2023-08-26T13:44:31.295 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local size=128 2023-08-26T13:44:31.295 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ '[' -n 128M ']' 2023-08-26T13:44:31.295 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ size=128M 2023-08-26T13:44:31.295 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ shift 2023-08-26T13:44:31.295 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock mirror/image_12 2023-08-26T13:44:31.336 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ '[' image = image ']' 2023-08-26T13:44:31.336 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ enable_mirror cluster2 mirror image_12 snapshot 2023-08-26T13:44:31.337 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster2 2023-08-26T13:44:31.337 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T13:44:31.337 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=image_12 2023-08-26T13:44:31.337 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local mode=snapshot 2023-08-26T13:44:31.337 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_12 snapshot 2023-08-26T13:44:32.126 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:Mirroring enabled 2023-08-26T13:44:32.134 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rbd --cluster=cluster2 info mirror/image_12 2023-08-26T13:44:32.171 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:rbd image 'image_12': 2023-08-26T13:44:32.172 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: size 128 MiB in 32 objects 2023-08-26T13:44:32.172 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: order 22 (4 MiB objects) 2023-08-26T13:44:32.172 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: snapshot_count: 1 2023-08-26T13:44:32.172 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: id: 12e5c625d078 2023-08-26T13:44:32.173 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: block_name_prefix: rbd_data.12e5c625d078 2023-08-26T13:44:32.173 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: format: 2 2023-08-26T13:44:32.173 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: features: layering, exclusive-lock 2023-08-26T13:44:32.173 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: op_features: 2023-08-26T13:44:32.173 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: flags: 2023-08-26T13:44:32.174 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: create_timestamp: Sat Aug 26 13:44:31 2023 2023-08-26T13:44:32.174 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: access_timestamp: Sat Aug 26 13:44:31 2023 2023-08-26T13:44:32.174 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: modify_timestamp: Sat Aug 26 13:44:31 2023 2023-08-26T13:44:32.174 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: mirroring state: enabled 2023-08-26T13:44:32.174 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: mirroring mode: snapshot 2023-08-26T13:44:32.175 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: mirroring global id: b189e689-d710-4185-9857-d4ed9dbb9df5 2023-08-26T13:44:32.175 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: mirroring primary: true 2023-08-26T13:44:32.176 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ '[' -n '' ']' 2023-08-26T13:44:32.176 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ stress_write_image cluster2 mirror image_12 2023-08-26T13:44:32.176 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster2 2023-08-26T13:44:32.176 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T13:44:32.176 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=image_12 2023-08-26T13:44:32.176 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2023-08-26T13:44:32.177 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local duration=11 2023-08-26T13:44:32.178 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ set +e 2023-08-26T13:44:32.178 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ timeout 11s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_12 --debug-rbd=20 --debug-journaler=20 2023-08-26T13:44:32.424 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:100 IOs, 62453760 bytes 2023-08-26T13:44:32.563 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:200 IOs, 89484288 bytes 2023-08-26T13:44:32.647 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:300 IOs, 90112000 bytes 2023-08-26T13:44:32.838 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:400 IOs, 135742464 bytes 2023-08-26T13:44:32.947 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:500 IOs, 147429376 bytes 2023-08-26T13:44:33.096 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:600 IOs, 181407744 bytes 2023-08-26T13:44:33.248 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:700 IOs, 214100992 bytes 2023-08-26T13:44:33.432 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:800 IOs, 273951744 bytes 2023-08-26T13:44:33.710 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:900 IOs, 352734208 bytes 2023-08-26T13:44:33.864 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:1000 IOs, 388435968 bytes 2023-08-26T13:44:33.984 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:1100 IOs, 416258048 bytes 2023-08-26T13:44:34.142 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:1200 IOs, 440381440 bytes 2023-08-26T13:44:34.315 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:1300 IOs, 480475136 bytes 2023-08-26T13:44:34.416 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:1400 IOs, 498486272 bytes 2023-08-26T13:44:34.635 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:1500 IOs, 560979968 bytes 2023-08-26T13:44:34.885 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:1600 IOs, 637306880 bytes 2023-08-26T13:44:35.044 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:1700 IOs, 673214464 bytes 2023-08-26T13:44:35.189 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:1800 IOs, 701477888 bytes 2023-08-26T13:44:35.407 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:1900 IOs, 762764288 bytes 2023-08-26T13:44:35.537 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:2000 IOs, 782593024 bytes 2023-08-26T13:44:35.660 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:2100 IOs, 800591872 bytes 2023-08-26T13:44:35.837 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:2200 IOs, 854814720 bytes 2023-08-26T13:44:36.041 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:2300 IOs, 901396480 bytes 2023-08-26T13:44:36.215 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:2400 IOs, 943072256 bytes 2023-08-26T13:44:36.611 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:2500 IOs, 1098768384 bytes 2023-08-26T13:44:36.723 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:2600 IOs, 1110568960 bytes 2023-08-26T13:44:36.884 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:2700 IOs, 1144516608 bytes 2023-08-26T13:44:37.145 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:2800 IOs, 1225916416 bytes 2023-08-26T13:44:37.426 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:2900 IOs, 1325427712 bytes 2023-08-26T13:44:37.586 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:3000 IOs, 1356036096 bytes 2023-08-26T13:44:37.727 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:3100 IOs, 1374033920 bytes 2023-08-26T13:44:37.966 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:3200 IOs, 1458016256 bytes 2023-08-26T13:44:38.138 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:3300 IOs, 1522882560 bytes 2023-08-26T13:44:38.340 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:3400 IOs, 1565029376 bytes 2023-08-26T13:44:38.532 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:3500 IOs, 1613336576 bytes 2023-08-26T13:44:38.633 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:3600 IOs, 1618625536 bytes 2023-08-26T13:44:38.778 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:3700 IOs, 1646281728 bytes 2023-08-26T13:44:38.922 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:3800 IOs, 1671919616 bytes 2023-08-26T13:44:39.128 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:3900 IOs, 1742110720 bytes 2023-08-26T13:44:39.386 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:4000 IOs, 1800414208 bytes 2023-08-26T13:44:39.719 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:4100 IOs, 1831758848 bytes 2023-08-26T13:44:39.858 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:4200 IOs, 1856190464 bytes 2023-08-26T13:44:40.081 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:4300 IOs, 1932249088 bytes 2023-08-26T13:44:40.270 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:4400 IOs, 1966144512 bytes 2023-08-26T13:44:40.364 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:4500 IOs, 1969823744 bytes 2023-08-26T13:44:40.478 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:4600 IOs, 1981668352 bytes 2023-08-26T13:44:40.682 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:4700 IOs, 2034626560 bytes 2023-08-26T13:44:40.917 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:4800 IOs, 2097837056 bytes 2023-08-26T13:44:41.061 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:4900 IOs, 2127529984 bytes 2023-08-26T13:44:41.164 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:5000 IOs, 2128143360 bytes 2023-08-26T13:44:41.313 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:5100 IOs, 2162035712 bytes 2023-08-26T13:44:41.495 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:5200 IOs, 2200574976 bytes 2023-08-26T13:44:41.630 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:5300 IOs, 2227856384 bytes 2023-08-26T13:44:41.737 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:5400 IOs, 2228465664 bytes 2023-08-26T13:44:41.979 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:5500 IOs, 2306392064 bytes 2023-08-26T13:44:42.188 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:5600 IOs, 2360787968 bytes 2023-08-26T13:44:42.384 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:5700 IOs, 2419075072 bytes 2023-08-26T13:44:42.557 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:5800 IOs, 2452998144 bytes 2023-08-26T13:44:42.647 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:5900 IOs, 2453612544 bytes 2023-08-26T13:44:42.851 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:6000 IOs, 2509968384 bytes 2023-08-26T13:44:43.041 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:6100 IOs, 2562104320 bytes 2023-08-26T13:44:43.181 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ error_code=124 2023-08-26T13:44:43.182 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ set -e 2023-08-26T13:44:43.182 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ '[' 124 -eq 124 ']' 2023-08-26T13:44:43.182 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ return 0 2023-08-26T13:44:43.182 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-08-26T13:44:43.183 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ image=image_13 2023-08-26T13:44:43.183 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ create_image_and_enable_mirror cluster2 mirror image_13 snapshot 128M 2023-08-26T13:44:43.183 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster2 2023-08-26T13:44:43.183 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ shift 2023-08-26T13:44:43.183 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T13:44:43.183 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ shift 2023-08-26T13:44:43.184 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=image_13 2023-08-26T13:44:43.184 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ shift 2023-08-26T13:44:43.184 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local mode=snapshot 2023-08-26T13:44:43.185 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ '[' -n snapshot ']' 2023-08-26T13:44:43.185 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ shift 2023-08-26T13:44:43.185 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ create_image cluster2 mirror image_13 128M 2023-08-26T13:44:43.185 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster2 2023-08-26T13:44:43.185 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ shift 2023-08-26T13:44:43.186 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T13:44:43.186 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ shift 2023-08-26T13:44:43.186 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=image_13 2023-08-26T13:44:43.186 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ shift 2023-08-26T13:44:43.186 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local size=128 2023-08-26T13:44:43.187 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ '[' -n 128M ']' 2023-08-26T13:44:43.187 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ size=128M 2023-08-26T13:44:43.187 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ shift 2023-08-26T13:44:43.187 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock mirror/image_13 2023-08-26T13:44:43.227 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ '[' image = image ']' 2023-08-26T13:44:43.227 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ enable_mirror cluster2 mirror image_13 snapshot 2023-08-26T13:44:43.228 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster2 2023-08-26T13:44:43.228 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T13:44:43.228 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=image_13 2023-08-26T13:44:43.228 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local mode=snapshot 2023-08-26T13:44:43.228 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_13 snapshot 2023-08-26T13:44:44.137 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:Mirroring enabled 2023-08-26T13:44:44.148 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rbd --cluster=cluster2 info mirror/image_13 2023-08-26T13:44:44.184 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:rbd image 'image_13': 2023-08-26T13:44:44.185 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: size 128 MiB in 32 objects 2023-08-26T13:44:44.185 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: order 22 (4 MiB objects) 2023-08-26T13:44:44.186 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: snapshot_count: 1 2023-08-26T13:44:44.186 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: id: 12ee4e133195 2023-08-26T13:44:44.186 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: block_name_prefix: rbd_data.12ee4e133195 2023-08-26T13:44:44.186 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: format: 2 2023-08-26T13:44:44.186 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: features: layering, exclusive-lock 2023-08-26T13:44:44.187 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: op_features: 2023-08-26T13:44:44.187 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: flags: 2023-08-26T13:44:44.187 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: create_timestamp: Sat Aug 26 13:44:43 2023 2023-08-26T13:44:44.187 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: access_timestamp: Sat Aug 26 13:44:43 2023 2023-08-26T13:44:44.188 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: modify_timestamp: Sat Aug 26 13:44:43 2023 2023-08-26T13:44:44.188 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: mirroring state: enabled 2023-08-26T13:44:44.188 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: mirroring mode: snapshot 2023-08-26T13:44:44.188 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: mirroring global id: 3f172be8-5137-4cda-906a-0c674ef9c674 2023-08-26T13:44:44.188 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: mirroring primary: true 2023-08-26T13:44:44.189 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ '[' -n '' ']' 2023-08-26T13:44:44.190 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ stress_write_image cluster2 mirror image_13 2023-08-26T13:44:44.190 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster2 2023-08-26T13:44:44.190 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T13:44:44.190 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=image_13 2023-08-26T13:44:44.190 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2023-08-26T13:44:44.191 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local duration=12 2023-08-26T13:44:44.191 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ set +e 2023-08-26T13:44:44.191 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ timeout 12s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_13 --debug-rbd=20 --debug-journaler=20 2023-08-26T13:44:44.480 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:100 IOs, 77809664 bytes 2023-08-26T13:44:44.952 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:200 IOs, 168493056 bytes 2023-08-26T13:44:45.066 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:300 IOs, 176452608 bytes 2023-08-26T13:44:45.219 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:400 IOs, 211619840 bytes 2023-08-26T13:44:45.357 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:500 IOs, 235267072 bytes 2023-08-26T13:44:45.460 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:600 IOs, 247988224 bytes 2023-08-26T13:44:45.672 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:700 IOs, 294754304 bytes 2023-08-26T13:44:45.848 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:800 IOs, 331020288 bytes 2023-08-26T13:44:45.959 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:900 IOs, 343424000 bytes 2023-08-26T13:44:46.117 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:1000 IOs, 393848832 bytes 2023-08-26T13:44:46.253 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:1100 IOs, 394465280 bytes 2023-08-26T13:44:46.474 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:1200 IOs, 475677696 bytes 2023-08-26T13:44:46.691 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:1300 IOs, 529114112 bytes 2023-08-26T13:44:46.797 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:1400 IOs, 540447744 bytes 2023-08-26T13:44:47.059 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:1500 IOs, 615980032 bytes 2023-08-26T13:44:47.216 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:1600 IOs, 662753280 bytes 2023-08-26T13:44:47.381 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:1700 IOs, 675133440 bytes 2023-08-26T13:44:47.667 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:1800 IOs, 769201152 bytes 2023-08-26T13:44:47.847 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:1900 IOs, 811443200 bytes 2023-08-26T13:44:47.979 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:2000 IOs, 832515072 bytes 2023-08-26T13:44:48.122 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:2100 IOs, 858713088 bytes 2023-08-26T13:44:48.269 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:2200 IOs, 888712192 bytes 2023-08-26T13:44:48.456 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:2300 IOs, 926614528 bytes 2023-08-26T13:44:48.736 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:2400 IOs, 1025143808 bytes 2023-08-26T13:44:48.921 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:2500 IOs, 1080713216 bytes 2023-08-26T13:44:49.042 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:2600 IOs, 1099853824 bytes 2023-08-26T13:44:49.245 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:2700 IOs, 1160200192 bytes 2023-08-26T13:44:49.420 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:2800 IOs, 1191327744 bytes 2023-08-26T13:44:49.592 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:2900 IOs, 1228134400 bytes 2023-08-26T13:44:49.775 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:3000 IOs, 1274212352 bytes 2023-08-26T13:44:49.965 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:3100 IOs, 1318389760 bytes 2023-08-26T13:44:50.186 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:3200 IOs, 1384755200 bytes 2023-08-26T13:44:50.352 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:3300 IOs, 1422740480 bytes 2023-08-26T13:44:50.462 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:3400 IOs, 1431777280 bytes 2023-08-26T13:44:50.672 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:3500 IOs, 1490045952 bytes 2023-08-26T13:44:50.789 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:3600 IOs, 1502493696 bytes 2023-08-26T13:44:50.938 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:3700 IOs, 1532570624 bytes 2023-08-26T13:44:51.085 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:3800 IOs, 1556068352 bytes 2023-08-26T13:44:51.182 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:3900 IOs, 1557303296 bytes 2023-08-26T13:44:51.336 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:4000 IOs, 1593102336 bytes 2023-08-26T13:44:51.530 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:4100 IOs, 1633195008 bytes 2023-08-26T13:44:51.731 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:4200 IOs, 1688686592 bytes 2023-08-26T13:44:51.828 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:4300 IOs, 1689303040 bytes 2023-08-26T13:44:51.984 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:4400 IOs, 1715751936 bytes 2023-08-26T13:44:52.134 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:4500 IOs, 1755462656 bytes 2023-08-26T13:44:52.394 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:4600 IOs, 1832836096 bytes 2023-08-26T13:44:52.608 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:4700 IOs, 1910371328 bytes 2023-08-26T13:44:53.034 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:4800 IOs, 2047413248 bytes 2023-08-26T13:44:53.154 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:4900 IOs, 2060173312 bytes 2023-08-26T13:44:53.353 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:5000 IOs, 2113511424 bytes 2023-08-26T13:44:53.483 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:5100 IOs, 2130516992 bytes 2023-08-26T13:44:53.763 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:5200 IOs, 2222436352 bytes 2023-08-26T13:44:53.897 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:5300 IOs, 2245498880 bytes 2023-08-26T13:44:54.105 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:5400 IOs, 2303446016 bytes 2023-08-26T13:44:54.247 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:5500 IOs, 2345391104 bytes 2023-08-26T13:44:54.482 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:5600 IOs, 2401580032 bytes 2023-08-26T13:44:54.698 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:5700 IOs, 2455777280 bytes 2023-08-26T13:44:54.906 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:5800 IOs, 2493255680 bytes 2023-08-26T13:44:55.040 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:5900 IOs, 2514708480 bytes 2023-08-26T13:44:55.273 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:6000 IOs, 2579400704 bytes 2023-08-26T13:44:55.580 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:6100 IOs, 2665284608 bytes 2023-08-26T13:44:55.772 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:6200 IOs, 2716273664 bytes 2023-08-26T13:44:55.902 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:6300 IOs, 2718372864 bytes 2023-08-26T13:44:56.059 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:6400 IOs, 2747394048 bytes 2023-08-26T13:44:56.196 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ error_code=124 2023-08-26T13:44:56.197 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ set -e 2023-08-26T13:44:56.197 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ '[' 124 -eq 124 ']' 2023-08-26T13:44:56.198 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ return 0 2023-08-26T13:44:56.198 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-08-26T13:44:56.198 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ image=image_14 2023-08-26T13:44:56.199 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ create_image_and_enable_mirror cluster2 mirror image_14 snapshot 128M 2023-08-26T13:44:56.199 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster2 2023-08-26T13:44:56.199 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ shift 2023-08-26T13:44:56.200 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T13:44:56.200 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ shift 2023-08-26T13:44:56.200 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=image_14 2023-08-26T13:44:56.200 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ shift 2023-08-26T13:44:56.200 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local mode=snapshot 2023-08-26T13:44:56.200 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ '[' -n snapshot ']' 2023-08-26T13:44:56.201 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ shift 2023-08-26T13:44:56.201 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ create_image cluster2 mirror image_14 128M 2023-08-26T13:44:56.201 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster2 2023-08-26T13:44:56.201 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ shift 2023-08-26T13:44:56.202 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T13:44:56.202 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ shift 2023-08-26T13:44:56.202 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=image_14 2023-08-26T13:44:56.202 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ shift 2023-08-26T13:44:56.202 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local size=128 2023-08-26T13:44:56.203 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ '[' -n 128M ']' 2023-08-26T13:44:56.203 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ size=128M 2023-08-26T13:44:56.203 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ shift 2023-08-26T13:44:56.203 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock mirror/image_14 2023-08-26T13:44:56.239 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ '[' image = image ']' 2023-08-26T13:44:56.240 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ enable_mirror cluster2 mirror image_14 snapshot 2023-08-26T13:44:56.240 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster2 2023-08-26T13:44:56.240 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T13:44:56.240 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=image_14 2023-08-26T13:44:56.240 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local mode=snapshot 2023-08-26T13:44:56.241 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_14 snapshot 2023-08-26T13:44:57.080 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:Mirroring enabled 2023-08-26T13:44:57.089 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rbd --cluster=cluster2 info mirror/image_14 2023-08-26T13:44:57.130 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:rbd image 'image_14': 2023-08-26T13:44:57.131 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: size 128 MiB in 32 objects 2023-08-26T13:44:57.131 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: order 22 (4 MiB objects) 2023-08-26T13:44:57.131 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: snapshot_count: 1 2023-08-26T13:44:57.131 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: id: 12f77196d702 2023-08-26T13:44:57.132 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: block_name_prefix: rbd_data.12f77196d702 2023-08-26T13:44:57.132 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: format: 2 2023-08-26T13:44:57.132 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: features: layering, exclusive-lock 2023-08-26T13:44:57.132 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: op_features: 2023-08-26T13:44:57.132 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: flags: 2023-08-26T13:44:57.133 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: create_timestamp: Sat Aug 26 13:44:56 2023 2023-08-26T13:44:57.133 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: access_timestamp: Sat Aug 26 13:44:56 2023 2023-08-26T13:44:57.133 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: modify_timestamp: Sat Aug 26 13:44:56 2023 2023-08-26T13:44:57.133 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: mirroring state: enabled 2023-08-26T13:44:57.133 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: mirroring mode: snapshot 2023-08-26T13:44:57.134 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: mirroring global id: 0fecd808-a8ef-45b7-aa0b-52385a8e2b1e 2023-08-26T13:44:57.134 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: mirroring primary: true 2023-08-26T13:44:57.135 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ '[' -n '' ']' 2023-08-26T13:44:57.135 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ stress_write_image cluster2 mirror image_14 2023-08-26T13:44:57.135 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster2 2023-08-26T13:44:57.135 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T13:44:57.135 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=image_14 2023-08-26T13:44:57.136 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2023-08-26T13:44:57.136 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local duration=11 2023-08-26T13:44:57.136 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ set +e 2023-08-26T13:44:57.137 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ timeout 11s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_14 --debug-rbd=20 --debug-journaler=20 2023-08-26T13:44:57.347 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:100 IOs, 31048704 bytes 2023-08-26T13:44:57.440 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:200 IOs, 31651840 bytes 2023-08-26T13:44:57.536 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:300 IOs, 37864448 bytes 2023-08-26T13:44:57.747 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:400 IOs, 101178368 bytes 2023-08-26T13:44:57.895 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:500 IOs, 122422272 bytes 2023-08-26T13:44:58.055 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:600 IOs, 153253888 bytes 2023-08-26T13:44:58.176 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:700 IOs, 167985152 bytes 2023-08-26T13:44:58.399 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:800 IOs, 226345984 bytes 2023-08-26T13:44:58.571 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:900 IOs, 287722496 bytes 2023-08-26T13:44:58.751 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:1000 IOs, 323038208 bytes 2023-08-26T13:44:58.920 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:1100 IOs, 346149888 bytes 2023-08-26T13:44:59.100 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:1200 IOs, 390604800 bytes 2023-08-26T13:44:59.257 INFO:tasks.ceph.cluster2.osd.0.smithi124.stderr:2023-08-26T13:44:59.256+0000 7fb60dfa7700 -1 reset not still connected to 0x56452ef86f70 2023-08-26T13:44:59.285 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:1300 IOs, 434810880 bytes 2023-08-26T13:44:59.491 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:1400 IOs, 476304384 bytes 2023-08-26T13:44:59.631 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:1500 IOs, 498697216 bytes 2023-08-26T13:44:59.842 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:1600 IOs, 572460032 bytes 2023-08-26T13:45:00.054 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:1700 IOs, 612136960 bytes 2023-08-26T13:45:00.182 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:1800 IOs, 632938496 bytes 2023-08-26T13:45:00.313 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:1900 IOs, 649174016 bytes 2023-08-26T13:45:00.452 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:2000 IOs, 672373760 bytes 2023-08-26T13:45:00.656 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:2100 IOs, 726507520 bytes 2023-08-26T13:45:00.900 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:2200 IOs, 805749760 bytes 2023-08-26T13:45:01.002 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:2300 IOs, 806356992 bytes 2023-08-26T13:45:01.109 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:2400 IOs, 815202304 bytes 2023-08-26T13:45:01.282 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:2500 IOs, 855339008 bytes 2023-08-26T13:45:01.422 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:2600 IOs, 874243072 bytes 2023-08-26T13:45:01.696 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:2700 IOs, 964752384 bytes 2023-08-26T13:45:01.921 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:2800 IOs, 1029914624 bytes 2023-08-26T13:45:02.067 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:2900 IOs, 1061216256 bytes 2023-08-26T13:45:02.267 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:3000 IOs, 1116795904 bytes 2023-08-26T13:45:02.410 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:3100 IOs, 1141493760 bytes 2023-08-26T13:45:02.598 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:3200 IOs, 1191856128 bytes 2023-08-26T13:45:02.790 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:3300 IOs, 1235703808 bytes 2023-08-26T13:45:02.949 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:3400 IOs, 1268682752 bytes 2023-08-26T13:45:03.123 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:3500 IOs, 1311314944 bytes 2023-08-26T13:45:03.254 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:3600 IOs, 1331236864 bytes 2023-08-26T13:45:03.445 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:3700 IOs, 1382626304 bytes 2023-08-26T13:45:03.611 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:3800 IOs, 1418074112 bytes 2023-08-26T13:45:03.935 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:3900 IOs, 1529267200 bytes 2023-08-26T13:45:04.041 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:4000 IOs, 1529886720 bytes 2023-08-26T13:45:04.218 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:4100 IOs, 1568727040 bytes 2023-08-26T13:45:04.434 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:4200 IOs, 1620921344 bytes 2023-08-26T13:45:04.608 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:4300 IOs, 1662146560 bytes 2023-08-26T13:45:04.731 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:4400 IOs, 1673609216 bytes 2023-08-26T13:45:04.884 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:4500 IOs, 1706578944 bytes 2023-08-26T13:45:05.063 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:4600 IOs, 1753666560 bytes 2023-08-26T13:45:05.223 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:4700 IOs, 1789532160 bytes 2023-08-26T13:45:05.404 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:4800 IOs, 1845692416 bytes 2023-08-26T13:45:05.658 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:4900 IOs, 1912101888 bytes 2023-08-26T13:45:05.869 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:5000 IOs, 1972761600 bytes 2023-08-26T13:45:06.113 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:5100 IOs, 2046842880 bytes 2023-08-26T13:45:06.307 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:5200 IOs, 2094874624 bytes 2023-08-26T13:45:06.417 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:5300 IOs, 2095523840 bytes 2023-08-26T13:45:06.731 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:5400 IOs, 2199291904 bytes 2023-08-26T13:45:06.878 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:5500 IOs, 2230733824 bytes 2023-08-26T13:45:07.014 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:5600 IOs, 2253017088 bytes 2023-08-26T13:45:07.218 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:5700 IOs, 2304076800 bytes 2023-08-26T13:45:07.380 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:5800 IOs, 2335638528 bytes 2023-08-26T13:45:07.586 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:5900 IOs, 2404086784 bytes 2023-08-26T13:45:07.727 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:6000 IOs, 2423375872 bytes 2023-08-26T13:45:07.949 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:6100 IOs, 2495976448 bytes 2023-08-26T13:45:08.128 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:6200 IOs, 2539181056 bytes 2023-08-26T13:45:08.142 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ error_code=124 2023-08-26T13:45:08.143 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ set -e 2023-08-26T13:45:08.143 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ '[' 124 -eq 124 ']' 2023-08-26T13:45:08.143 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ return 0 2023-08-26T13:45:08.143 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-08-26T13:45:08.144 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ image=image_15 2023-08-26T13:45:08.144 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ create_image_and_enable_mirror cluster2 mirror image_15 snapshot 128M 2023-08-26T13:45:08.144 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster2 2023-08-26T13:45:08.144 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ shift 2023-08-26T13:45:08.144 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T13:45:08.144 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ shift 2023-08-26T13:45:08.145 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=image_15 2023-08-26T13:45:08.145 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ shift 2023-08-26T13:45:08.145 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local mode=snapshot 2023-08-26T13:45:08.145 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ '[' -n snapshot ']' 2023-08-26T13:45:08.145 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ shift 2023-08-26T13:45:08.146 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ create_image cluster2 mirror image_15 128M 2023-08-26T13:45:08.146 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster2 2023-08-26T13:45:08.146 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ shift 2023-08-26T13:45:08.146 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T13:45:08.146 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ shift 2023-08-26T13:45:08.147 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=image_15 2023-08-26T13:45:08.147 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ shift 2023-08-26T13:45:08.147 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local size=128 2023-08-26T13:45:08.147 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ '[' -n 128M ']' 2023-08-26T13:45:08.147 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ size=128M 2023-08-26T13:45:08.148 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ shift 2023-08-26T13:45:08.148 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock mirror/image_15 2023-08-26T13:45:08.186 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ '[' image = image ']' 2023-08-26T13:45:08.186 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ enable_mirror cluster2 mirror image_15 snapshot 2023-08-26T13:45:08.187 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster2 2023-08-26T13:45:08.187 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T13:45:08.187 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=image_15 2023-08-26T13:45:08.187 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local mode=snapshot 2023-08-26T13:45:08.187 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_15 snapshot 2023-08-26T13:45:09.083 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:Mirroring enabled 2023-08-26T13:45:09.092 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rbd --cluster=cluster2 info mirror/image_15 2023-08-26T13:45:09.132 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:rbd image 'image_15': 2023-08-26T13:45:09.132 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: size 128 MiB in 32 objects 2023-08-26T13:45:09.132 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: order 22 (4 MiB objects) 2023-08-26T13:45:09.132 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: snapshot_count: 1 2023-08-26T13:45:09.132 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: id: 130023fccbce 2023-08-26T13:45:09.133 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: block_name_prefix: rbd_data.130023fccbce 2023-08-26T13:45:09.133 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: format: 2 2023-08-26T13:45:09.133 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: features: layering, exclusive-lock 2023-08-26T13:45:09.133 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: op_features: 2023-08-26T13:45:09.133 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: flags: 2023-08-26T13:45:09.134 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: create_timestamp: Sat Aug 26 13:45:08 2023 2023-08-26T13:45:09.134 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: access_timestamp: Sat Aug 26 13:45:08 2023 2023-08-26T13:45:09.134 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: modify_timestamp: Sat Aug 26 13:45:08 2023 2023-08-26T13:45:09.134 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: mirroring state: enabled 2023-08-26T13:45:09.134 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: mirroring mode: snapshot 2023-08-26T13:45:09.135 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: mirroring global id: 57b5e313-8920-4ded-8ea4-ab852f9e42c2 2023-08-26T13:45:09.135 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: mirroring primary: true 2023-08-26T13:45:09.136 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ '[' -n '' ']' 2023-08-26T13:45:09.136 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ stress_write_image cluster2 mirror image_15 2023-08-26T13:45:09.136 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster2 2023-08-26T13:45:09.136 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T13:45:09.136 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=image_15 2023-08-26T13:45:09.137 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2023-08-26T13:45:09.137 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local duration=6 2023-08-26T13:45:09.138 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ set +e 2023-08-26T13:45:09.138 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ timeout 6s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_15 --debug-rbd=20 --debug-journaler=20 2023-08-26T13:45:09.339 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:100 IOs, 38975488 bytes 2023-08-26T13:45:09.430 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:200 IOs, 41236480 bytes 2023-08-26T13:45:09.642 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:300 IOs, 113613824 bytes 2023-08-26T13:45:09.845 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:400 IOs, 169753600 bytes 2023-08-26T13:45:09.993 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:500 IOs, 193738752 bytes 2023-08-26T13:45:10.125 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:600 IOs, 205729792 bytes 2023-08-26T13:45:10.261 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:700 IOs, 231304192 bytes 2023-08-26T13:45:10.544 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:800 IOs, 319799296 bytes 2023-08-26T13:45:10.709 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:900 IOs, 359762944 bytes 2023-08-26T13:45:10.892 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:1000 IOs, 420346880 bytes 2023-08-26T13:45:11.053 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:1100 IOs, 436915200 bytes 2023-08-26T13:45:11.207 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:1200 IOs, 463898624 bytes 2023-08-26T13:45:11.297 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:1300 IOs, 464515072 bytes 2023-08-26T13:45:11.469 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:1400 IOs, 502476800 bytes 2023-08-26T13:45:11.670 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:1500 IOs, 573959168 bytes 2023-08-26T13:45:11.860 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:1600 IOs, 598522880 bytes 2023-08-26T13:45:12.090 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:1700 IOs, 663983104 bytes 2023-08-26T13:45:12.267 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:1800 IOs, 707210240 bytes 2023-08-26T13:45:12.481 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:1900 IOs, 785366016 bytes 2023-08-26T13:45:12.654 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:2000 IOs, 811753472 bytes 2023-08-26T13:45:12.774 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:2100 IOs, 824693760 bytes 2023-08-26T13:45:12.991 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:2200 IOs, 891998208 bytes 2023-08-26T13:45:13.155 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:2300 IOs, 944006144 bytes 2023-08-26T13:45:13.317 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:2400 IOs, 960894976 bytes 2023-08-26T13:45:13.575 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:2500 IOs, 1042808832 bytes 2023-08-26T13:45:13.778 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:2600 IOs, 1101898752 bytes 2023-08-26T13:45:13.875 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:2700 IOs, 1102502912 bytes 2023-08-26T13:45:13.966 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:2800 IOs, 1106180096 bytes 2023-08-26T13:45:14.137 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:2900 IOs, 1141039104 bytes 2023-08-26T13:45:14.334 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:3000 IOs, 1194139648 bytes 2023-08-26T13:45:14.464 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:3100 IOs, 1212503040 bytes 2023-08-26T13:45:14.667 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:3200 IOs, 1270178816 bytes 2023-08-26T13:45:14.811 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:3300 IOs, 1292128256 bytes 2023-08-26T13:45:14.999 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:3400 IOs, 1343995904 bytes 2023-08-26T13:45:15.144 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ error_code=124 2023-08-26T13:45:15.144 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ set -e 2023-08-26T13:45:15.144 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ '[' 124 -eq 124 ']' 2023-08-26T13:45:15.144 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ return 0 2023-08-26T13:45:15.145 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-08-26T13:45:15.145 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ image=image_16 2023-08-26T13:45:15.146 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ create_image_and_enable_mirror cluster2 mirror image_16 snapshot 128M 2023-08-26T13:45:15.147 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster2 2023-08-26T13:45:15.147 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ shift 2023-08-26T13:45:15.147 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T13:45:15.147 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ shift 2023-08-26T13:45:15.147 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=image_16 2023-08-26T13:45:15.148 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ shift 2023-08-26T13:45:15.148 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local mode=snapshot 2023-08-26T13:45:15.148 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ '[' -n snapshot ']' 2023-08-26T13:45:15.148 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ shift 2023-08-26T13:45:15.148 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ create_image cluster2 mirror image_16 128M 2023-08-26T13:45:15.149 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster2 2023-08-26T13:45:15.149 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ shift 2023-08-26T13:45:15.149 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T13:45:15.149 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ shift 2023-08-26T13:45:15.149 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=image_16 2023-08-26T13:45:15.149 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ shift 2023-08-26T13:45:15.150 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local size=128 2023-08-26T13:45:15.150 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ '[' -n 128M ']' 2023-08-26T13:45:15.150 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ size=128M 2023-08-26T13:45:15.150 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ shift 2023-08-26T13:45:15.150 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock mirror/image_16 2023-08-26T13:45:15.190 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ '[' image = image ']' 2023-08-26T13:45:15.191 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ enable_mirror cluster2 mirror image_16 snapshot 2023-08-26T13:45:15.191 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster2 2023-08-26T13:45:15.191 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T13:45:15.191 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=image_16 2023-08-26T13:45:15.191 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local mode=snapshot 2023-08-26T13:45:15.192 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_16 snapshot 2023-08-26T13:45:15.893 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:Mirroring enabled 2023-08-26T13:45:15.900 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rbd --cluster=cluster2 info mirror/image_16 2023-08-26T13:45:15.936 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:rbd image 'image_16': 2023-08-26T13:45:15.937 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: size 128 MiB in 32 objects 2023-08-26T13:45:15.937 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: order 22 (4 MiB objects) 2023-08-26T13:45:15.937 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: snapshot_count: 1 2023-08-26T13:45:15.937 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: id: 1309f2599bfb 2023-08-26T13:45:15.938 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: block_name_prefix: rbd_data.1309f2599bfb 2023-08-26T13:45:15.938 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: format: 2 2023-08-26T13:45:15.938 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: features: layering, exclusive-lock 2023-08-26T13:45:15.938 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: op_features: 2023-08-26T13:45:15.938 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: flags: 2023-08-26T13:45:15.939 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: create_timestamp: Sat Aug 26 13:45:15 2023 2023-08-26T13:45:15.939 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: access_timestamp: Sat Aug 26 13:45:15 2023 2023-08-26T13:45:15.939 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: modify_timestamp: Sat Aug 26 13:45:15 2023 2023-08-26T13:45:15.939 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: mirroring state: enabled 2023-08-26T13:45:15.940 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: mirroring mode: snapshot 2023-08-26T13:45:15.940 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: mirroring global id: bd80dfca-7cec-4de6-be52-c58244873bd3 2023-08-26T13:45:15.940 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: mirroring primary: true 2023-08-26T13:45:15.941 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ '[' -n '' ']' 2023-08-26T13:45:15.941 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ stress_write_image cluster2 mirror image_16 2023-08-26T13:45:15.941 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster2 2023-08-26T13:45:15.942 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T13:45:15.942 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=image_16 2023-08-26T13:45:15.942 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2023-08-26T13:45:15.942 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local duration=8 2023-08-26T13:45:15.942 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ set +e 2023-08-26T13:45:15.942 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ timeout 8s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_16 --debug-rbd=20 --debug-journaler=20 2023-08-26T13:45:16.117 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:100 IOs, 18969600 bytes 2023-08-26T13:45:16.362 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:200 IOs, 101963776 bytes 2023-08-26T13:45:16.506 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:300 IOs, 123443200 bytes 2023-08-26T13:45:16.616 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:400 IOs, 133985280 bytes 2023-08-26T13:45:16.885 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:500 IOs, 225156096 bytes 2023-08-26T13:45:17.077 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:600 IOs, 276727808 bytes 2023-08-26T13:45:17.226 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:700 IOs, 311065600 bytes 2023-08-26T13:45:17.344 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:800 IOs, 323061760 bytes 2023-08-26T13:45:17.495 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:900 IOs, 352160768 bytes 2023-08-26T13:45:17.594 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:1000 IOs, 355020800 bytes 2023-08-26T13:45:17.707 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:1100 IOs, 357517312 bytes 2023-08-26T13:45:17.926 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:1200 IOs, 418329600 bytes 2023-08-26T13:45:18.099 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:1300 IOs, 455126016 bytes 2023-08-26T13:45:18.427 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:1400 IOs, 575529984 bytes 2023-08-26T13:45:18.586 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:1500 IOs, 627408896 bytes 2023-08-26T13:45:18.738 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:1600 IOs, 646982656 bytes 2023-08-26T13:45:18.990 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:1700 IOs, 737997824 bytes 2023-08-26T13:45:19.111 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:1800 IOs, 752253952 bytes 2023-08-26T13:45:19.247 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:1900 IOs, 776175616 bytes 2023-08-26T13:45:19.410 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:2000 IOs, 812835840 bytes 2023-08-26T13:45:19.648 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:2100 IOs, 883109888 bytes 2023-08-26T13:45:19.799 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:2200 IOs, 919155712 bytes 2023-08-26T13:45:19.991 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:2300 IOs, 973882368 bytes 2023-08-26T13:45:20.237 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:2400 IOs, 1044578304 bytes 2023-08-26T13:45:20.390 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:2500 IOs, 1076039680 bytes 2023-08-26T13:45:20.538 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:2600 IOs, 1104576512 bytes 2023-08-26T13:45:20.723 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:2700 IOs, 1146973184 bytes 2023-08-26T13:45:20.849 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:2800 IOs, 1162210304 bytes 2023-08-26T13:45:21.140 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:2900 IOs, 1264507904 bytes 2023-08-26T13:45:21.277 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:3000 IOs, 1289978880 bytes 2023-08-26T13:45:21.430 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:3100 IOs, 1329633280 bytes 2023-08-26T13:45:21.651 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:3200 IOs, 1399856128 bytes 2023-08-26T13:45:21.757 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:3300 IOs, 1400469504 bytes 2023-08-26T13:45:21.971 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:3400 IOs, 1465443328 bytes 2023-08-26T13:45:22.153 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:3500 IOs, 1512163328 bytes 2023-08-26T13:45:22.370 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:3600 IOs, 1579197440 bytes 2023-08-26T13:45:22.548 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:3700 IOs, 1616350208 bytes 2023-08-26T13:45:22.698 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:3800 IOs, 1642176512 bytes 2023-08-26T13:45:22.851 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:3900 IOs, 1671382016 bytes 2023-08-26T13:45:22.975 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:4000 IOs, 1688018944 bytes 2023-08-26T13:45:23.142 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:4100 IOs, 1726206976 bytes 2023-08-26T13:45:23.359 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:4200 IOs, 1785612288 bytes 2023-08-26T13:45:23.559 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:4300 IOs, 1849630720 bytes 2023-08-26T13:45:23.723 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:4400 IOs, 1871234048 bytes 2023-08-26T13:45:23.925 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:4500 IOs, 1920411648 bytes 2023-08-26T13:45:23.949 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ error_code=124 2023-08-26T13:45:23.950 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ set -e 2023-08-26T13:45:23.950 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ '[' 124 -eq 124 ']' 2023-08-26T13:45:23.950 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ return 0 2023-08-26T13:45:23.951 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-08-26T13:45:23.951 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ image=image_17 2023-08-26T13:45:23.951 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ create_image_and_enable_mirror cluster2 mirror image_17 snapshot 128M 2023-08-26T13:45:23.952 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster2 2023-08-26T13:45:23.952 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ shift 2023-08-26T13:45:23.952 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T13:45:23.952 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ shift 2023-08-26T13:45:23.953 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=image_17 2023-08-26T13:45:23.953 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ shift 2023-08-26T13:45:23.953 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local mode=snapshot 2023-08-26T13:45:23.953 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ '[' -n snapshot ']' 2023-08-26T13:45:23.954 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ shift 2023-08-26T13:45:23.954 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ create_image cluster2 mirror image_17 128M 2023-08-26T13:45:23.955 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster2 2023-08-26T13:45:23.955 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ shift 2023-08-26T13:45:23.955 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T13:45:23.955 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ shift 2023-08-26T13:45:23.956 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=image_17 2023-08-26T13:45:23.956 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ shift 2023-08-26T13:45:23.956 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local size=128 2023-08-26T13:45:23.956 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ '[' -n 128M ']' 2023-08-26T13:45:23.956 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ size=128M 2023-08-26T13:45:23.956 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ shift 2023-08-26T13:45:23.957 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock mirror/image_17 2023-08-26T13:45:23.994 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ '[' image = image ']' 2023-08-26T13:45:23.995 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ enable_mirror cluster2 mirror image_17 snapshot 2023-08-26T13:45:23.995 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster2 2023-08-26T13:45:23.996 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T13:45:23.996 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=image_17 2023-08-26T13:45:23.996 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local mode=snapshot 2023-08-26T13:45:23.997 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_17 snapshot 2023-08-26T13:45:24.944 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:Mirroring enabled 2023-08-26T13:45:24.953 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rbd --cluster=cluster2 info mirror/image_17 2023-08-26T13:45:24.994 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:rbd image 'image_17': 2023-08-26T13:45:24.995 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: size 128 MiB in 32 objects 2023-08-26T13:45:24.995 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: order 22 (4 MiB objects) 2023-08-26T13:45:24.995 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: snapshot_count: 1 2023-08-26T13:45:24.995 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: id: 13117453d656 2023-08-26T13:45:24.995 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: block_name_prefix: rbd_data.13117453d656 2023-08-26T13:45:24.996 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: format: 2 2023-08-26T13:45:24.996 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: features: layering, exclusive-lock 2023-08-26T13:45:24.996 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: op_features: 2023-08-26T13:45:24.996 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: flags: 2023-08-26T13:45:24.997 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: create_timestamp: Sat Aug 26 13:45:23 2023 2023-08-26T13:45:24.997 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: access_timestamp: Sat Aug 26 13:45:23 2023 2023-08-26T13:45:24.997 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: modify_timestamp: Sat Aug 26 13:45:23 2023 2023-08-26T13:45:24.997 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: mirroring state: enabled 2023-08-26T13:45:24.997 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: mirroring mode: snapshot 2023-08-26T13:45:24.998 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: mirroring global id: ec21e764-3783-49f1-b802-01400de8810c 2023-08-26T13:45:24.998 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: mirroring primary: true 2023-08-26T13:45:24.999 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ '[' -n '' ']' 2023-08-26T13:45:24.999 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ stress_write_image cluster2 mirror image_17 2023-08-26T13:45:25.000 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster2 2023-08-26T13:45:25.000 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T13:45:25.000 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=image_17 2023-08-26T13:45:25.000 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2023-08-26T13:45:25.001 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local duration=11 2023-08-26T13:45:25.001 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ set +e 2023-08-26T13:45:25.001 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ timeout 11s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_17 --debug-rbd=20 --debug-journaler=20 2023-08-26T13:45:25.280 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:100 IOs, 66902016 bytes 2023-08-26T13:45:25.480 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:200 IOs, 83968000 bytes 2023-08-26T13:45:25.613 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:300 IOs, 99387392 bytes 2023-08-26T13:45:25.758 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:400 IOs, 127163392 bytes 2023-08-26T13:45:25.954 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:500 IOs, 179165184 bytes 2023-08-26T13:45:26.064 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:600 IOs, 194438144 bytes 2023-08-26T13:45:26.368 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:700 IOs, 299679744 bytes 2023-08-26T13:45:26.528 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:800 IOs, 330485760 bytes 2023-08-26T13:45:26.713 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:900 IOs, 373149696 bytes 2023-08-26T13:45:26.954 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:1000 IOs, 452333568 bytes 2023-08-26T13:45:27.211 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:1100 IOs, 542256128 bytes 2023-08-26T13:45:27.388 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:1200 IOs, 568921088 bytes 2023-08-26T13:45:27.622 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:1300 IOs, 647487488 bytes 2023-08-26T13:45:27.884 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:1400 IOs, 727884800 bytes 2023-08-26T13:45:28.009 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:1500 IOs, 739612672 bytes 2023-08-26T13:45:28.196 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:1600 IOs, 806478848 bytes 2023-08-26T13:45:28.424 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:1700 IOs, 865829888 bytes 2023-08-26T13:45:28.622 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:1800 IOs, 914382848 bytes 2023-08-26T13:45:28.808 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:1900 IOs, 957602816 bytes 2023-08-26T13:45:28.931 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:2000 IOs, 966000640 bytes 2023-08-26T13:45:29.102 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:2100 IOs, 1012088832 bytes 2023-08-26T13:45:29.275 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:2200 IOs, 1062028288 bytes 2023-08-26T13:45:29.488 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:2300 IOs, 1113700352 bytes 2023-08-26T13:45:29.679 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:2400 IOs, 1151579136 bytes 2023-08-26T13:45:29.792 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:2500 IOs, 1160630272 bytes 2023-08-26T13:45:30.020 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:2600 IOs, 1223435264 bytes 2023-08-26T13:45:30.229 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:2700 IOs, 1277821952 bytes 2023-08-26T13:45:30.332 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:2800 IOs, 1278420992 bytes 2023-08-26T13:45:30.593 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:2900 IOs, 1361965056 bytes 2023-08-26T13:45:30.692 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:3000 IOs, 1362568192 bytes 2023-08-26T13:45:30.798 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:3100 IOs, 1363751936 bytes 2023-08-26T13:45:31.003 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:3200 IOs, 1420590080 bytes 2023-08-26T13:45:31.173 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:3300 IOs, 1456129024 bytes 2023-08-26T13:45:31.328 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:3400 IOs, 1488223232 bytes 2023-08-26T13:45:31.546 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:3500 IOs, 1550917632 bytes 2023-08-26T13:45:31.664 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:3600 IOs, 1564676096 bytes 2023-08-26T13:45:31.845 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:3700 IOs, 1598588928 bytes 2023-08-26T13:45:32.042 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:3800 IOs, 1650953216 bytes 2023-08-26T13:45:32.215 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:3900 IOs, 1690598400 bytes 2023-08-26T13:45:32.321 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:4000 IOs, 1691222016 bytes 2023-08-26T13:45:32.460 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:4100 IOs, 1716636672 bytes 2023-08-26T13:45:32.595 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:4200 IOs, 1739889664 bytes 2023-08-26T13:45:32.722 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:4300 IOs, 1751406592 bytes 2023-08-26T13:45:32.928 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:4400 IOs, 1828911104 bytes 2023-08-26T13:45:33.066 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:4500 IOs, 1829547008 bytes 2023-08-26T13:45:33.350 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:4600 IOs, 1920096256 bytes 2023-08-26T13:45:33.485 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:4700 IOs, 1936848896 bytes 2023-08-26T13:45:33.594 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:4800 IOs, 1960108032 bytes 2023-08-26T13:45:33.816 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:4900 IOs, 2002569216 bytes 2023-08-26T13:45:33.960 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:5000 IOs, 2026660864 bytes 2023-08-26T13:45:34.155 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:5100 IOs, 2096437248 bytes 2023-08-26T13:45:34.396 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:5200 IOs, 2152090624 bytes 2023-08-26T13:45:34.530 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:5300 IOs, 2164470784 bytes 2023-08-26T13:45:34.744 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:5400 IOs, 2226514944 bytes 2023-08-26T13:45:34.911 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:5500 IOs, 2282423296 bytes 2023-08-26T13:45:35.033 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:5600 IOs, 2283065344 bytes 2023-08-26T13:45:35.256 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:5700 IOs, 2370912256 bytes 2023-08-26T13:45:35.391 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:5800 IOs, 2371526656 bytes 2023-08-26T13:45:35.563 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:5900 IOs, 2419348480 bytes 2023-08-26T13:45:35.685 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:6000 IOs, 2432189440 bytes 2023-08-26T13:45:35.820 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:6100 IOs, 2453700608 bytes 2023-08-26T13:45:36.007 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ error_code=124 2023-08-26T13:45:36.007 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ set -e 2023-08-26T13:45:36.008 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ '[' 124 -eq 124 ']' 2023-08-26T13:45:36.008 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ return 0 2023-08-26T13:45:36.009 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-08-26T13:45:36.009 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ image=image_18 2023-08-26T13:45:36.009 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ create_image_and_enable_mirror cluster2 mirror image_18 snapshot 128M 2023-08-26T13:45:36.010 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster2 2023-08-26T13:45:36.010 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ shift 2023-08-26T13:45:36.010 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T13:45:36.011 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ shift 2023-08-26T13:45:36.011 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=image_18 2023-08-26T13:45:36.011 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ shift 2023-08-26T13:45:36.012 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local mode=snapshot 2023-08-26T13:45:36.012 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ '[' -n snapshot ']' 2023-08-26T13:45:36.012 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ shift 2023-08-26T13:45:36.013 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ create_image cluster2 mirror image_18 128M 2023-08-26T13:45:36.013 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster2 2023-08-26T13:45:36.014 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ shift 2023-08-26T13:45:36.014 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T13:45:36.014 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ shift 2023-08-26T13:45:36.015 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=image_18 2023-08-26T13:45:36.015 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ shift 2023-08-26T13:45:36.015 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local size=128 2023-08-26T13:45:36.016 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ '[' -n 128M ']' 2023-08-26T13:45:36.016 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ size=128M 2023-08-26T13:45:36.016 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ shift 2023-08-26T13:45:36.017 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock mirror/image_18 2023-08-26T13:45:36.052 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ '[' image = image ']' 2023-08-26T13:45:36.053 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ enable_mirror cluster2 mirror image_18 snapshot 2023-08-26T13:45:36.053 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster2 2023-08-26T13:45:36.053 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T13:45:36.054 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=image_18 2023-08-26T13:45:36.054 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local mode=snapshot 2023-08-26T13:45:36.054 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_18 snapshot 2023-08-26T13:45:37.088 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:Mirroring enabled 2023-08-26T13:45:37.097 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rbd --cluster=cluster2 info mirror/image_18 2023-08-26T13:45:37.136 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:rbd image 'image_18': 2023-08-26T13:45:37.136 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: size 128 MiB in 32 objects 2023-08-26T13:45:37.136 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: order 22 (4 MiB objects) 2023-08-26T13:45:37.137 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: snapshot_count: 1 2023-08-26T13:45:37.137 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: id: 1319678d303b 2023-08-26T13:45:37.137 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: block_name_prefix: rbd_data.1319678d303b 2023-08-26T13:45:37.137 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: format: 2 2023-08-26T13:45:37.137 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: features: layering, exclusive-lock 2023-08-26T13:45:37.138 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: op_features: 2023-08-26T13:45:37.138 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: flags: 2023-08-26T13:45:37.138 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: create_timestamp: Sat Aug 26 13:45:36 2023 2023-08-26T13:45:37.138 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: access_timestamp: Sat Aug 26 13:45:36 2023 2023-08-26T13:45:37.138 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: modify_timestamp: Sat Aug 26 13:45:36 2023 2023-08-26T13:45:37.139 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: mirroring state: enabled 2023-08-26T13:45:37.139 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: mirroring mode: snapshot 2023-08-26T13:45:37.139 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: mirroring global id: 5293326c-bb2a-4d36-bfa9-7cfd4da48fe1 2023-08-26T13:45:37.139 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: mirroring primary: true 2023-08-26T13:45:37.140 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ '[' -n '' ']' 2023-08-26T13:45:37.140 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ stress_write_image cluster2 mirror image_18 2023-08-26T13:45:37.140 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster2 2023-08-26T13:45:37.141 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T13:45:37.141 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=image_18 2023-08-26T13:45:37.141 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2023-08-26T13:45:37.141 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local duration=5 2023-08-26T13:45:37.142 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ set +e 2023-08-26T13:45:37.142 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ timeout 5s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_18 --debug-rbd=20 --debug-journaler=20 2023-08-26T13:45:37.386 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:100 IOs, 54097920 bytes 2023-08-26T13:45:37.584 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:200 IOs, 102545408 bytes 2023-08-26T13:45:37.772 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:300 IOs, 151416832 bytes 2023-08-26T13:45:37.906 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:400 IOs, 182264832 bytes 2023-08-26T13:45:38.153 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:500 IOs, 266267648 bytes 2023-08-26T13:45:38.520 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:600 IOs, 407112704 bytes 2023-08-26T13:45:38.763 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:700 IOs, 495201280 bytes 2023-08-26T13:45:38.997 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:800 IOs, 551868416 bytes 2023-08-26T13:45:39.172 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:900 IOs, 595287040 bytes 2023-08-26T13:45:39.372 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:1000 IOs, 641715200 bytes 2023-08-26T13:45:39.506 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:1100 IOs, 662017024 bytes 2023-08-26T13:45:39.692 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:1200 IOs, 710478848 bytes 2023-08-26T13:45:39.872 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:1300 IOs, 755120128 bytes 2023-08-26T13:45:40.018 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:1400 IOs, 786700288 bytes 2023-08-26T13:45:40.123 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:1500 IOs, 787309568 bytes 2023-08-26T13:45:40.345 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:1600 IOs, 851433472 bytes 2023-08-26T13:45:40.514 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:1700 IOs, 885114880 bytes 2023-08-26T13:45:40.654 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:1800 IOs, 929073152 bytes 2023-08-26T13:45:40.877 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:1900 IOs, 981810176 bytes 2023-08-26T13:45:41.104 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:2000 IOs, 1051887616 bytes 2023-08-26T13:45:41.238 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:2100 IOs, 1084694528 bytes 2023-08-26T13:45:41.499 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:2200 IOs, 1158054912 bytes 2023-08-26T13:45:41.696 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:2300 IOs, 1206367232 bytes 2023-08-26T13:45:41.839 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:2400 IOs, 1246009344 bytes 2023-08-26T13:45:42.115 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:2500 IOs, 1325450240 bytes 2023-08-26T13:45:42.146 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ error_code=124 2023-08-26T13:45:42.147 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ set -e 2023-08-26T13:45:42.147 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ '[' 124 -eq 124 ']' 2023-08-26T13:45:42.147 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ return 0 2023-08-26T13:45:42.147 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-08-26T13:45:42.147 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ image=image_19 2023-08-26T13:45:42.148 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ create_image_and_enable_mirror cluster2 mirror image_19 snapshot 128M 2023-08-26T13:45:42.148 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster2 2023-08-26T13:45:42.148 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ shift 2023-08-26T13:45:42.148 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T13:45:42.148 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ shift 2023-08-26T13:45:42.149 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=image_19 2023-08-26T13:45:42.149 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ shift 2023-08-26T13:45:42.149 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local mode=snapshot 2023-08-26T13:45:42.149 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ '[' -n snapshot ']' 2023-08-26T13:45:42.149 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ shift 2023-08-26T13:45:42.150 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ create_image cluster2 mirror image_19 128M 2023-08-26T13:45:42.150 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster2 2023-08-26T13:45:42.150 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ shift 2023-08-26T13:45:42.150 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T13:45:42.150 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ shift 2023-08-26T13:45:42.150 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=image_19 2023-08-26T13:45:42.151 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ shift 2023-08-26T13:45:42.151 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local size=128 2023-08-26T13:45:42.151 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ '[' -n 128M ']' 2023-08-26T13:45:42.151 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ size=128M 2023-08-26T13:45:42.151 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ shift 2023-08-26T13:45:42.152 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock mirror/image_19 2023-08-26T13:45:42.193 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ '[' image = image ']' 2023-08-26T13:45:42.193 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ enable_mirror cluster2 mirror image_19 snapshot 2023-08-26T13:45:42.193 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster2 2023-08-26T13:45:42.193 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T13:45:42.194 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=image_19 2023-08-26T13:45:42.194 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local mode=snapshot 2023-08-26T13:45:42.194 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_19 snapshot 2023-08-26T13:45:43.132 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:Mirroring enabled 2023-08-26T13:45:43.145 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rbd --cluster=cluster2 info mirror/image_19 2023-08-26T13:45:43.184 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:rbd image 'image_19': 2023-08-26T13:45:43.184 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: size 128 MiB in 32 objects 2023-08-26T13:45:43.184 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: order 22 (4 MiB objects) 2023-08-26T13:45:43.184 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: snapshot_count: 1 2023-08-26T13:45:43.185 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: id: 132491bdc96f 2023-08-26T13:45:43.185 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: block_name_prefix: rbd_data.132491bdc96f 2023-08-26T13:45:43.185 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: format: 2 2023-08-26T13:45:43.185 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: features: layering, exclusive-lock 2023-08-26T13:45:43.186 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: op_features: 2023-08-26T13:45:43.186 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: flags: 2023-08-26T13:45:43.186 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: create_timestamp: Sat Aug 26 13:45:42 2023 2023-08-26T13:45:43.186 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: access_timestamp: Sat Aug 26 13:45:42 2023 2023-08-26T13:45:43.186 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: modify_timestamp: Sat Aug 26 13:45:42 2023 2023-08-26T13:45:43.187 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: mirroring state: enabled 2023-08-26T13:45:43.187 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: mirroring mode: snapshot 2023-08-26T13:45:43.187 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: mirroring global id: f1b3d4dd-7b2d-44de-a0df-388999259524 2023-08-26T13:45:43.187 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: mirroring primary: true 2023-08-26T13:45:43.189 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ '[' -n '' ']' 2023-08-26T13:45:43.189 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ stress_write_image cluster2 mirror image_19 2023-08-26T13:45:43.189 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster2 2023-08-26T13:45:43.189 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T13:45:43.190 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=image_19 2023-08-26T13:45:43.190 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2023-08-26T13:45:43.190 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local duration=9 2023-08-26T13:45:43.190 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ set +e 2023-08-26T13:45:43.190 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ timeout 9s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_19 --debug-rbd=20 --debug-journaler=20 2023-08-26T13:45:43.355 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:100 IOs, 24871936 bytes 2023-08-26T13:45:43.492 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:200 IOs, 41396224 bytes 2023-08-26T13:45:43.625 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:300 IOs, 60408832 bytes 2023-08-26T13:45:43.828 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:400 IOs, 114708480 bytes 2023-08-26T13:45:43.940 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:500 IOs, 126290944 bytes 2023-08-26T13:45:44.115 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:600 IOs, 170347520 bytes 2023-08-26T13:45:44.224 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:700 IOs, 170962944 bytes 2023-08-26T13:45:44.373 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:800 IOs, 196401152 bytes 2023-08-26T13:45:44.481 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:900 IOs, 215971840 bytes 2023-08-26T13:45:44.727 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:1000 IOs, 285083648 bytes 2023-08-26T13:45:44.908 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:1100 IOs, 336884736 bytes 2023-08-26T13:45:45.021 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:1200 IOs, 337479680 bytes 2023-08-26T13:45:45.130 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:1300 IOs, 353122304 bytes 2023-08-26T13:45:45.306 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:1400 IOs, 391817216 bytes 2023-08-26T13:45:45.440 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:1500 IOs, 406710272 bytes 2023-08-26T13:45:45.581 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:1600 IOs, 434548736 bytes 2023-08-26T13:45:45.787 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:1700 IOs, 489128960 bytes 2023-08-26T13:45:46.046 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:1800 IOs, 579262464 bytes 2023-08-26T13:45:46.214 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:1900 IOs, 619431936 bytes 2023-08-26T13:45:46.515 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:2000 IOs, 725579776 bytes 2023-08-26T13:45:46.780 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:2100 IOs, 818838528 bytes 2023-08-26T13:45:46.950 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:2200 IOs, 855913472 bytes 2023-08-26T13:45:47.164 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:2300 IOs, 913335296 bytes 2023-08-26T13:45:47.257 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:2400 IOs, 914299904 bytes 2023-08-26T13:45:47.487 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:2500 IOs, 986257408 bytes 2023-08-26T13:45:47.682 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:2600 IOs, 1035891712 bytes 2023-08-26T13:45:47.796 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:2700 IOs, 1042863104 bytes 2023-08-26T13:45:48.020 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:2800 IOs, 1106887680 bytes 2023-08-26T13:45:48.157 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:2900 IOs, 1137136640 bytes 2023-08-26T13:45:48.345 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:3000 IOs, 1182022656 bytes 2023-08-26T13:45:48.512 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:3100 IOs, 1219720192 bytes 2023-08-26T13:45:48.695 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:3200 IOs, 1261571072 bytes 2023-08-26T13:45:48.789 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:3300 IOs, 1262164992 bytes 2023-08-26T13:45:49.079 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:3400 IOs, 1362894848 bytes 2023-08-26T13:45:49.417 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:3500 IOs, 1489201152 bytes 2023-08-26T13:45:49.574 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:3600 IOs, 1512017920 bytes 2023-08-26T13:45:49.676 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:3700 IOs, 1518870528 bytes 2023-08-26T13:45:49.875 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:3800 IOs, 1568935936 bytes 2023-08-26T13:45:50.041 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:3900 IOs, 1605295104 bytes 2023-08-26T13:45:50.222 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:4000 IOs, 1645534208 bytes 2023-08-26T13:45:50.353 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:4100 IOs, 1664404480 bytes 2023-08-26T13:45:50.501 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:4200 IOs, 1692953600 bytes 2023-08-26T13:45:50.746 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:4300 IOs, 1778585600 bytes 2023-08-26T13:45:50.872 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:4400 IOs, 1798401024 bytes 2023-08-26T13:45:51.133 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:4500 IOs, 1896987648 bytes 2023-08-26T13:45:51.411 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:4600 IOs, 1985705984 bytes 2023-08-26T13:45:51.603 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.0 with skip (0.5095546313707008) > weight (0.25) 2023-08-26T13:45:51.603 INFO:tasks.rbd_mirror_thrash:kill cluster1.client.mirror.1 2023-08-26T13:45:51.604 INFO:tasks.rbd_mirror.cluster1.client.mirror.1:Sent signal 15 2023-08-26T13:45:51.604 INFO:tasks.rbd_mirror_thrash:waiting for 0 secs before reviving daemons 2023-08-26T13:45:51.604 INFO:tasks.rbd_mirror_thrash:waiting for cluster1.client.mirror.1 2023-08-26T13:45:51.604 INFO:teuthology.orchestra.run:waiting for 600 2023-08-26T13:45:51.605 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi124.stderr:2023-08-26T13:45:51.604+0000 7ff32135f700 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.1 (PID: 111136) UID: 1000 2023-08-26T13:45:51.607 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:4700 IOs, 2052505600 bytes 2023-08-26T13:45:51.840 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:4800 IOs, 2142606336 bytes 2023-08-26T13:45:52.163 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:4900 IOs, 2226219008 bytes 2023-08-26T13:45:52.195 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ error_code=124 2023-08-26T13:45:52.195 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ set -e 2023-08-26T13:45:52.195 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ '[' 124 -eq 124 ']' 2023-08-26T13:45:52.196 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ return 0 2023-08-26T13:45:52.196 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-08-26T13:45:52.196 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ image=image_20 2023-08-26T13:45:52.196 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ create_image_and_enable_mirror cluster2 mirror image_20 snapshot 128M 2023-08-26T13:45:52.196 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster2 2023-08-26T13:45:52.197 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ shift 2023-08-26T13:45:52.197 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T13:45:52.197 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ shift 2023-08-26T13:45:52.197 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=image_20 2023-08-26T13:45:52.197 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ shift 2023-08-26T13:45:52.197 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local mode=snapshot 2023-08-26T13:45:52.198 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ '[' -n snapshot ']' 2023-08-26T13:45:52.198 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ shift 2023-08-26T13:45:52.198 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ create_image cluster2 mirror image_20 128M 2023-08-26T13:45:52.198 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster2 2023-08-26T13:45:52.198 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ shift 2023-08-26T13:45:52.199 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T13:45:52.199 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ shift 2023-08-26T13:45:52.199 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=image_20 2023-08-26T13:45:52.199 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ shift 2023-08-26T13:45:52.199 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local size=128 2023-08-26T13:45:52.200 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ '[' -n 128M ']' 2023-08-26T13:45:52.200 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ size=128M 2023-08-26T13:45:52.200 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ shift 2023-08-26T13:45:52.200 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock mirror/image_20 2023-08-26T13:45:52.241 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ '[' image = image ']' 2023-08-26T13:45:52.241 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ enable_mirror cluster2 mirror image_20 snapshot 2023-08-26T13:45:52.242 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster2 2023-08-26T13:45:52.242 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T13:45:52.242 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=image_20 2023-08-26T13:45:52.242 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local mode=snapshot 2023-08-26T13:45:52.242 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_20 snapshot 2023-08-26T13:45:53.219 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:Mirroring enabled 2023-08-26T13:45:53.227 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rbd --cluster=cluster2 info mirror/image_20 2023-08-26T13:45:53.265 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:rbd image 'image_20': 2023-08-26T13:45:53.265 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: size 128 MiB in 32 objects 2023-08-26T13:45:53.266 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: order 22 (4 MiB objects) 2023-08-26T13:45:53.266 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: snapshot_count: 1 2023-08-26T13:45:53.266 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: id: 132dfd2cf456 2023-08-26T13:45:53.266 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: block_name_prefix: rbd_data.132dfd2cf456 2023-08-26T13:45:53.266 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: format: 2 2023-08-26T13:45:53.267 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: features: layering, exclusive-lock 2023-08-26T13:45:53.267 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: op_features: 2023-08-26T13:45:53.267 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: flags: 2023-08-26T13:45:53.267 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: create_timestamp: Sat Aug 26 13:45:52 2023 2023-08-26T13:45:53.268 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: access_timestamp: Sat Aug 26 13:45:52 2023 2023-08-26T13:45:53.268 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: modify_timestamp: Sat Aug 26 13:45:52 2023 2023-08-26T13:45:53.268 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: mirroring state: enabled 2023-08-26T13:45:53.268 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: mirroring mode: snapshot 2023-08-26T13:45:53.268 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: mirroring global id: e8a98866-d970-4e86-bb3b-2de28303ceff 2023-08-26T13:45:53.268 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: mirroring primary: true 2023-08-26T13:45:53.269 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ '[' -n '' ']' 2023-08-26T13:45:53.270 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ stress_write_image cluster2 mirror image_20 2023-08-26T13:45:53.270 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster2 2023-08-26T13:45:53.270 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T13:45:53.270 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=image_20 2023-08-26T13:45:53.270 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2023-08-26T13:45:53.271 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local duration=12 2023-08-26T13:45:53.271 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ set +e 2023-08-26T13:45:53.271 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ timeout 12s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_20 --debug-rbd=20 --debug-journaler=20 2023-08-26T13:45:53.444 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:100 IOs, 12329984 bytes 2023-08-26T13:45:53.582 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:200 IOs, 47643648 bytes 2023-08-26T13:45:53.771 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:300 IOs, 86218752 bytes 2023-08-26T13:45:53.913 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:400 IOs, 107133952 bytes 2023-08-26T13:45:54.148 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:500 IOs, 178705408 bytes 2023-08-26T13:45:54.297 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:600 IOs, 206624768 bytes 2023-08-26T13:45:54.512 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:700 IOs, 235669504 bytes 2023-08-26T13:45:54.872 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:800 IOs, 347006976 bytes 2023-08-26T13:45:55.177 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:900 IOs, 463088640 bytes 2023-08-26T13:45:55.301 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:1000 IOs, 480715776 bytes 2023-08-26T13:45:55.444 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:1100 IOs, 489760768 bytes 2023-08-26T13:45:55.605 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:1200 IOs, 537700352 bytes 2023-08-26T13:45:55.782 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:1300 IOs, 567249920 bytes 2023-08-26T13:45:55.984 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:1400 IOs, 618547200 bytes 2023-08-26T13:45:56.193 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:1500 IOs, 695664640 bytes 2023-08-26T13:45:56.461 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:1600 IOs, 754889728 bytes 2023-08-26T13:45:56.682 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:1700 IOs, 815712256 bytes 2023-08-26T13:45:56.845 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:1800 IOs, 849105920 bytes 2023-08-26T13:45:57.192 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:1900 IOs, 972187648 bytes 2023-08-26T13:45:57.336 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:2000 IOs, 997951488 bytes 2023-08-26T13:45:57.480 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:2100 IOs, 1032160256 bytes 2023-08-26T13:45:57.663 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:2200 IOs, 1057421312 bytes 2023-08-26T13:45:57.706 INFO:tasks.rbd_mirror_thrash:reviving cluster1.client.mirror.1 2023-08-26T13:45:57.706 INFO:tasks.rbd_mirror.cluster1.client.mirror.1:Restarting daemon 2023-08-26T13:45:57.706 DEBUG:teuthology.orchestra.run.smithi124:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.1 2023-08-26T13:45:57.715 INFO:tasks.rbd_mirror.cluster1.client.mirror.1:Started 2023-08-26T13:45:57.715 INFO:tasks.rbd_mirror_thrash:waiting for 83 secs before thrashing 2023-08-26T13:45:57.888 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:2300 IOs, 1110084608 bytes 2023-08-26T13:45:58.200 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:2400 IOs, 1231006720 bytes 2023-08-26T13:45:58.422 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:2500 IOs, 1308848128 bytes 2023-08-26T13:45:58.603 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:2600 IOs, 1331032064 bytes 2023-08-26T13:45:58.799 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:2700 IOs, 1378591744 bytes 2023-08-26T13:45:58.981 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:2800 IOs, 1420178432 bytes 2023-08-26T13:45:59.146 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:2900 IOs, 1451897856 bytes 2023-08-26T13:45:59.251 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:3000 IOs, 1470019584 bytes 2023-08-26T13:45:59.394 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:3100 IOs, 1476542464 bytes 2023-08-26T13:45:59.569 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:3200 IOs, 1519416320 bytes 2023-08-26T13:45:59.858 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:3300 IOs, 1607016448 bytes 2023-08-26T13:46:00.030 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:3400 IOs, 1639839744 bytes 2023-08-26T13:46:00.220 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:3500 IOs, 1686504448 bytes 2023-08-26T13:46:00.412 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:3600 IOs, 1748654080 bytes 2023-08-26T13:46:00.585 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:3700 IOs, 1780731904 bytes 2023-08-26T13:46:00.686 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:3800 IOs, 1781342208 bytes 2023-08-26T13:46:00.887 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:3900 IOs, 1836099584 bytes 2023-08-26T13:46:01.099 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:4000 IOs, 1898936320 bytes 2023-08-26T13:46:01.204 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:4100 IOs, 1904607232 bytes 2023-08-26T13:46:01.399 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:4200 IOs, 1962821632 bytes 2023-08-26T13:46:01.619 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:4300 IOs, 2025210880 bytes 2023-08-26T13:46:01.831 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:4400 IOs, 2084619264 bytes 2023-08-26T13:46:02.009 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:4500 IOs, 2122207232 bytes 2023-08-26T13:46:02.270 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:4600 IOs, 2191325184 bytes 2023-08-26T13:46:02.391 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:4700 IOs, 2204135424 bytes 2023-08-26T13:46:02.547 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:4800 IOs, 2234106880 bytes 2023-08-26T13:46:02.752 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:4900 IOs, 2278040576 bytes 2023-08-26T13:46:02.896 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:5000 IOs, 2300267520 bytes 2023-08-26T13:46:03.297 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:5100 IOs, 2371713024 bytes 2023-08-26T13:46:03.508 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:5200 IOs, 2432360448 bytes 2023-08-26T13:46:03.676 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:5300 IOs, 2446683136 bytes 2023-08-26T13:46:03.845 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:5400 IOs, 2487134208 bytes 2023-08-26T13:46:04.054 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:5500 IOs, 2546860032 bytes 2023-08-26T13:46:04.196 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:5600 IOs, 2577118208 bytes 2023-08-26T13:46:04.447 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:5700 IOs, 2649824256 bytes 2023-08-26T13:46:04.636 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:5800 IOs, 2699522048 bytes 2023-08-26T13:46:04.823 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:5900 IOs, 2743493632 bytes 2023-08-26T13:46:05.013 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:6000 IOs, 2791838720 bytes 2023-08-26T13:46:05.233 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:6100 IOs, 2866331648 bytes 2023-08-26T13:46:05.279 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ error_code=124 2023-08-26T13:46:05.280 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ set -e 2023-08-26T13:46:05.280 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ '[' 124 -eq 124 ']' 2023-08-26T13:46:05.280 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ return 0 2023-08-26T13:46:05.280 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-08-26T13:46:05.281 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ image=image_21 2023-08-26T13:46:05.281 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ create_image_and_enable_mirror cluster2 mirror image_21 snapshot 128M 2023-08-26T13:46:05.281 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster2 2023-08-26T13:46:05.281 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ shift 2023-08-26T13:46:05.281 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T13:46:05.282 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ shift 2023-08-26T13:46:05.282 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=image_21 2023-08-26T13:46:05.282 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ shift 2023-08-26T13:46:05.282 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local mode=snapshot 2023-08-26T13:46:05.282 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ '[' -n snapshot ']' 2023-08-26T13:46:05.283 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ shift 2023-08-26T13:46:05.283 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ create_image cluster2 mirror image_21 128M 2023-08-26T13:46:05.283 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster2 2023-08-26T13:46:05.283 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ shift 2023-08-26T13:46:05.283 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T13:46:05.283 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ shift 2023-08-26T13:46:05.284 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=image_21 2023-08-26T13:46:05.284 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ shift 2023-08-26T13:46:05.284 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local size=128 2023-08-26T13:46:05.284 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ '[' -n 128M ']' 2023-08-26T13:46:05.284 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ size=128M 2023-08-26T13:46:05.285 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ shift 2023-08-26T13:46:05.285 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock mirror/image_21 2023-08-26T13:46:05.324 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ '[' image = image ']' 2023-08-26T13:46:05.324 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ enable_mirror cluster2 mirror image_21 snapshot 2023-08-26T13:46:05.324 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster2 2023-08-26T13:46:05.325 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T13:46:05.325 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=image_21 2023-08-26T13:46:05.325 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local mode=snapshot 2023-08-26T13:46:05.325 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_21 snapshot 2023-08-26T13:46:05.950 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:Mirroring enabled 2023-08-26T13:46:05.957 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rbd --cluster=cluster2 info mirror/image_21 2023-08-26T13:46:05.992 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:rbd image 'image_21': 2023-08-26T13:46:05.993 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: size 128 MiB in 32 objects 2023-08-26T13:46:05.993 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: order 22 (4 MiB objects) 2023-08-26T13:46:05.993 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: snapshot_count: 1 2023-08-26T13:46:05.993 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: id: 133eb005a2b 2023-08-26T13:46:05.993 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: block_name_prefix: rbd_data.133eb005a2b 2023-08-26T13:46:05.994 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: format: 2 2023-08-26T13:46:05.994 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: features: layering, exclusive-lock 2023-08-26T13:46:05.994 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: op_features: 2023-08-26T13:46:05.994 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: flags: 2023-08-26T13:46:05.994 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: create_timestamp: Sat Aug 26 13:46:05 2023 2023-08-26T13:46:05.995 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: access_timestamp: Sat Aug 26 13:46:05 2023 2023-08-26T13:46:05.995 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: modify_timestamp: Sat Aug 26 13:46:05 2023 2023-08-26T13:46:05.995 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: mirroring state: enabled 2023-08-26T13:46:05.995 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: mirroring mode: snapshot 2023-08-26T13:46:05.996 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: mirroring global id: 04903506-ddd2-48b7-835f-3797d11e35d3 2023-08-26T13:46:05.996 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: mirroring primary: true 2023-08-26T13:46:05.997 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ '[' -n '' ']' 2023-08-26T13:46:05.998 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ stress_write_image cluster2 mirror image_21 2023-08-26T13:46:05.998 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster2 2023-08-26T13:46:05.998 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T13:46:05.998 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=image_21 2023-08-26T13:46:05.998 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2023-08-26T13:46:05.999 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local duration=7 2023-08-26T13:46:05.999 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ set +e 2023-08-26T13:46:05.999 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ timeout 7s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_21 --debug-rbd=20 --debug-journaler=20 2023-08-26T13:46:06.203 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:100 IOs, 36957184 bytes 2023-08-26T13:46:06.437 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:200 IOs, 102871040 bytes 2023-08-26T13:46:06.646 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:300 IOs, 158116864 bytes 2023-08-26T13:46:06.743 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:400 IOs, 158743552 bytes 2023-08-26T13:46:06.971 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:500 IOs, 230378496 bytes 2023-08-26T13:46:07.104 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:600 IOs, 255150080 bytes 2023-08-26T13:46:07.271 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:700 IOs, 285392896 bytes 2023-08-26T13:46:07.367 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:800 IOs, 287319040 bytes 2023-08-26T13:46:07.551 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:900 IOs, 332601344 bytes 2023-08-26T13:46:07.727 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:1000 IOs, 374290432 bytes 2023-08-26T13:46:07.845 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:1100 IOs, 383726592 bytes 2023-08-26T13:46:07.995 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:1200 IOs, 424932352 bytes 2023-08-26T13:46:08.121 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:1300 IOs, 425533440 bytes 2023-08-26T13:46:08.374 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:1400 IOs, 502423552 bytes 2023-08-26T13:46:08.580 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:1500 IOs, 563039232 bytes 2023-08-26T13:46:08.678 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:1600 IOs, 566753280 bytes 2023-08-26T13:46:08.787 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:1700 IOs, 567397376 bytes 2023-08-26T13:46:08.962 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:1800 IOs, 613388288 bytes 2023-08-26T13:46:09.166 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:1900 IOs, 655446016 bytes 2023-08-26T13:46:09.260 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:2000 IOs, 656068608 bytes 2023-08-26T13:46:09.438 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:2100 IOs, 709476352 bytes 2023-08-26T13:46:09.692 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:2200 IOs, 764831744 bytes 2023-08-26T13:46:09.961 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:2300 IOs, 847789056 bytes 2023-08-26T13:46:10.176 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:2400 IOs, 898275328 bytes 2023-08-26T13:46:10.302 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:2500 IOs, 916603904 bytes 2023-08-26T13:46:10.407 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:2600 IOs, 917244928 bytes 2023-08-26T13:46:10.493 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.0 with skip (0.9291900084922762) > weight (0.25) 2023-08-26T13:46:10.494 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.1 with skip (0.6163064861571949) > weight (0.25) 2023-08-26T13:46:10.494 INFO:tasks.rbd_mirror_thrash:kill cluster2.client.mirror.2 2023-08-26T13:46:10.494 INFO:tasks.rbd_mirror.cluster2.client.mirror.2:Sent signal 15 2023-08-26T13:46:10.495 INFO:tasks.rbd_mirror_thrash:waiting for 9 secs before reviving daemons 2023-08-26T13:46:10.495 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi124.stderr:2023-08-26T13:46:10.493+0000 7f2f3048b700 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.2 (PID: 106119) UID: 1000 2023-08-26T13:46:10.598 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:2700 IOs, 964997120 bytes 2023-08-26T13:46:10.779 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:2800 IOs, 997303296 bytes 2023-08-26T13:46:10.874 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:2900 IOs, 997929984 bytes 2023-08-26T13:46:11.040 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:3000 IOs, 1033771008 bytes 2023-08-26T13:46:11.177 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:3100 IOs, 1054876672 bytes 2023-08-26T13:46:11.442 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:3200 IOs, 1149285376 bytes 2023-08-26T13:46:11.732 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:3300 IOs, 1239793664 bytes 2023-08-26T13:46:12.025 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:3400 IOs, 1351081984 bytes 2023-08-26T13:46:12.288 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:3500 IOs, 1424621568 bytes 2023-08-26T13:46:12.467 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:3600 IOs, 1467234304 bytes 2023-08-26T13:46:12.863 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:3700 IOs, 1603036160 bytes 2023-08-26T13:46:13.005 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ error_code=124 2023-08-26T13:46:13.005 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ set -e 2023-08-26T13:46:13.006 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ '[' 124 -eq 124 ']' 2023-08-26T13:46:13.006 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ return 0 2023-08-26T13:46:13.006 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-08-26T13:46:13.006 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ image=image_22 2023-08-26T13:46:13.006 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ create_image_and_enable_mirror cluster2 mirror image_22 snapshot 128M 2023-08-26T13:46:13.006 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster2 2023-08-26T13:46:13.007 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ shift 2023-08-26T13:46:13.007 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T13:46:13.007 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ shift 2023-08-26T13:46:13.007 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=image_22 2023-08-26T13:46:13.007 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ shift 2023-08-26T13:46:13.008 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local mode=snapshot 2023-08-26T13:46:13.008 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ '[' -n snapshot ']' 2023-08-26T13:46:13.008 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ shift 2023-08-26T13:46:13.008 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ create_image cluster2 mirror image_22 128M 2023-08-26T13:46:13.008 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster2 2023-08-26T13:46:13.009 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ shift 2023-08-26T13:46:13.009 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T13:46:13.009 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ shift 2023-08-26T13:46:13.009 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=image_22 2023-08-26T13:46:13.009 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ shift 2023-08-26T13:46:13.009 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local size=128 2023-08-26T13:46:13.010 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ '[' -n 128M ']' 2023-08-26T13:46:13.010 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ size=128M 2023-08-26T13:46:13.010 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ shift 2023-08-26T13:46:13.010 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock mirror/image_22 2023-08-26T13:46:13.052 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ '[' image = image ']' 2023-08-26T13:46:13.053 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ enable_mirror cluster2 mirror image_22 snapshot 2023-08-26T13:46:13.053 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster2 2023-08-26T13:46:13.053 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T13:46:13.053 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=image_22 2023-08-26T13:46:13.053 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local mode=snapshot 2023-08-26T13:46:13.054 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_22 snapshot 2023-08-26T13:46:13.154 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:Mirroring enabled 2023-08-26T13:46:13.162 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rbd --cluster=cluster2 info mirror/image_22 2023-08-26T13:46:13.197 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:rbd image 'image_22': 2023-08-26T13:46:13.197 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: size 128 MiB in 32 objects 2023-08-26T13:46:13.198 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: order 22 (4 MiB objects) 2023-08-26T13:46:13.198 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: snapshot_count: 1 2023-08-26T13:46:13.198 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: id: 1348b95cc0d1 2023-08-26T13:46:13.198 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: block_name_prefix: rbd_data.1348b95cc0d1 2023-08-26T13:46:13.198 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: format: 2 2023-08-26T13:46:13.199 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: features: layering, exclusive-lock 2023-08-26T13:46:13.199 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: op_features: 2023-08-26T13:46:13.199 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: flags: 2023-08-26T13:46:13.199 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: create_timestamp: Sat Aug 26 13:46:13 2023 2023-08-26T13:46:13.199 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: access_timestamp: Sat Aug 26 13:46:13 2023 2023-08-26T13:46:13.200 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: modify_timestamp: Sat Aug 26 13:46:13 2023 2023-08-26T13:46:13.200 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: mirroring state: enabled 2023-08-26T13:46:13.200 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: mirroring mode: snapshot 2023-08-26T13:46:13.200 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: mirroring global id: 70414d4b-2367-4c05-8614-971399383919 2023-08-26T13:46:13.200 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: mirroring primary: true 2023-08-26T13:46:13.201 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ '[' -n '' ']' 2023-08-26T13:46:13.201 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ stress_write_image cluster2 mirror image_22 2023-08-26T13:46:13.202 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster2 2023-08-26T13:46:13.202 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T13:46:13.202 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=image_22 2023-08-26T13:46:13.202 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2023-08-26T13:46:13.203 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local duration=7 2023-08-26T13:46:13.203 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ set +e 2023-08-26T13:46:13.203 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ timeout 7s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_22 --debug-rbd=20 --debug-journaler=20 2023-08-26T13:46:13.534 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:100 IOs, 131097600 bytes 2023-08-26T13:46:13.677 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:200 IOs, 136297472 bytes 2023-08-26T13:46:13.775 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:300 IOs, 136899584 bytes 2023-08-26T13:46:13.904 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:400 IOs, 155127808 bytes 2023-08-26T13:46:14.025 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:500 IOs, 183909376 bytes 2023-08-26T13:46:14.337 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:600 IOs, 284576768 bytes 2023-08-26T13:46:14.462 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:700 IOs, 296022016 bytes 2023-08-26T13:46:14.582 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:800 IOs, 306957312 bytes 2023-08-26T13:46:14.692 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:900 IOs, 312423424 bytes 2023-08-26T13:46:14.879 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:1000 IOs, 354239488 bytes 2023-08-26T13:46:15.044 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:1100 IOs, 389891072 bytes 2023-08-26T13:46:15.154 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:1200 IOs, 394810368 bytes 2023-08-26T13:46:15.374 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:1300 IOs, 455355392 bytes 2023-08-26T13:46:15.601 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:1400 IOs, 528978944 bytes 2023-08-26T13:46:15.835 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:1500 IOs, 587507712 bytes 2023-08-26T13:46:16.024 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:1600 IOs, 636507136 bytes 2023-08-26T13:46:16.177 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:1700 IOs, 669393920 bytes 2023-08-26T13:46:16.416 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:1800 IOs, 739641344 bytes 2023-08-26T13:46:16.634 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:1900 IOs, 800005120 bytes 2023-08-26T13:46:16.830 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:2000 IOs, 849290240 bytes 2023-08-26T13:46:16.918 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:2100 IOs, 849887232 bytes 2023-08-26T13:46:17.065 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:2200 IOs, 887476224 bytes 2023-08-26T13:46:17.345 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:2300 IOs, 975657984 bytes 2023-08-26T13:46:17.496 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:2400 IOs, 1005847552 bytes 2023-08-26T13:46:17.691 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:2500 IOs, 1048006656 bytes 2023-08-26T13:46:17.930 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:2600 IOs, 1137189888 bytes 2023-08-26T13:46:18.098 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:2700 IOs, 1149612032 bytes 2023-08-26T13:46:18.382 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:2800 IOs, 1247566848 bytes 2023-08-26T13:46:18.546 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:2900 IOs, 1278594048 bytes 2023-08-26T13:46:18.706 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:3000 IOs, 1305335808 bytes 2023-08-26T13:46:18.833 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:3100 IOs, 1318235136 bytes 2023-08-26T13:46:19.017 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:3200 IOs, 1364218880 bytes 2023-08-26T13:46:19.193 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:3300 IOs, 1409804288 bytes 2023-08-26T13:46:19.485 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:3400 IOs, 1512226816 bytes 2023-08-26T13:46:19.495 INFO:tasks.rbd_mirror_thrash:waiting for cluster2.client.mirror.2 2023-08-26T13:46:19.496 INFO:teuthology.orchestra.run:waiting for 600 2023-08-26T13:46:19.496 INFO:tasks.rbd_mirror_thrash:reviving cluster2.client.mirror.2 2023-08-26T13:46:19.496 INFO:tasks.rbd_mirror.cluster2.client.mirror.2:Restarting daemon 2023-08-26T13:46:19.497 DEBUG:teuthology.orchestra.run.smithi124:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.2 2023-08-26T13:46:19.499 INFO:tasks.rbd_mirror.cluster2.client.mirror.2:Started 2023-08-26T13:46:19.500 INFO:tasks.rbd_mirror_thrash:waiting for 97 secs before thrashing 2023-08-26T13:46:19.630 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:3500 IOs, 1528783872 bytes 2023-08-26T13:46:19.795 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:3600 IOs, 1553934336 bytes 2023-08-26T13:46:19.931 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:3700 IOs, 1566141440 bytes 2023-08-26T13:46:20.099 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:3800 IOs, 1588432896 bytes 2023-08-26T13:46:20.209 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ error_code=124 2023-08-26T13:46:20.209 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ set -e 2023-08-26T13:46:20.209 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ '[' 124 -eq 124 ']' 2023-08-26T13:46:20.210 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ return 0 2023-08-26T13:46:20.210 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-08-26T13:46:20.210 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ image=image_23 2023-08-26T13:46:20.210 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ create_image_and_enable_mirror cluster2 mirror image_23 snapshot 128M 2023-08-26T13:46:20.210 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster2 2023-08-26T13:46:20.211 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ shift 2023-08-26T13:46:20.211 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T13:46:20.211 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ shift 2023-08-26T13:46:20.211 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=image_23 2023-08-26T13:46:20.211 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ shift 2023-08-26T13:46:20.212 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local mode=snapshot 2023-08-26T13:46:20.212 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ '[' -n snapshot ']' 2023-08-26T13:46:20.212 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ shift 2023-08-26T13:46:20.212 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ create_image cluster2 mirror image_23 128M 2023-08-26T13:46:20.212 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster2 2023-08-26T13:46:20.213 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ shift 2023-08-26T13:46:20.213 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T13:46:20.213 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ shift 2023-08-26T13:46:20.213 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=image_23 2023-08-26T13:46:20.213 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ shift 2023-08-26T13:46:20.213 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local size=128 2023-08-26T13:46:20.214 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ '[' -n 128M ']' 2023-08-26T13:46:20.214 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ size=128M 2023-08-26T13:46:20.214 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ shift 2023-08-26T13:46:20.214 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock mirror/image_23 2023-08-26T13:46:20.257 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ '[' image = image ']' 2023-08-26T13:46:20.257 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ enable_mirror cluster2 mirror image_23 snapshot 2023-08-26T13:46:20.257 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster2 2023-08-26T13:46:20.258 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T13:46:20.258 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=image_23 2023-08-26T13:46:20.258 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local mode=snapshot 2023-08-26T13:46:20.258 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_23 snapshot 2023-08-26T13:46:21.200 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:Mirroring enabled 2023-08-26T13:46:21.207 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rbd --cluster=cluster2 info mirror/image_23 2023-08-26T13:46:21.246 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:rbd image 'image_23': 2023-08-26T13:46:21.246 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: size 128 MiB in 32 objects 2023-08-26T13:46:21.247 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: order 22 (4 MiB objects) 2023-08-26T13:46:21.247 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: snapshot_count: 1 2023-08-26T13:46:21.247 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: id: 135c3b2632ed 2023-08-26T13:46:21.247 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: block_name_prefix: rbd_data.135c3b2632ed 2023-08-26T13:46:21.247 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: format: 2 2023-08-26T13:46:21.248 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: features: layering, exclusive-lock 2023-08-26T13:46:21.248 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: op_features: 2023-08-26T13:46:21.248 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: flags: 2023-08-26T13:46:21.248 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: create_timestamp: Sat Aug 26 13:46:20 2023 2023-08-26T13:46:21.248 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: access_timestamp: Sat Aug 26 13:46:20 2023 2023-08-26T13:46:21.248 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: modify_timestamp: Sat Aug 26 13:46:20 2023 2023-08-26T13:46:21.249 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: mirroring state: enabled 2023-08-26T13:46:21.249 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: mirroring mode: snapshot 2023-08-26T13:46:21.249 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: mirroring global id: ba978cfa-fbe5-4a3a-86fa-f20310f2d99f 2023-08-26T13:46:21.249 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: mirroring primary: true 2023-08-26T13:46:21.250 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ '[' -n '' ']' 2023-08-26T13:46:21.250 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ stress_write_image cluster2 mirror image_23 2023-08-26T13:46:21.250 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster2 2023-08-26T13:46:21.251 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T13:46:21.251 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=image_23 2023-08-26T13:46:21.251 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2023-08-26T13:46:21.252 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local duration=8 2023-08-26T13:46:21.252 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ set +e 2023-08-26T13:46:21.252 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ timeout 8s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_23 --debug-rbd=20 --debug-journaler=20 2023-08-26T13:46:21.414 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:100 IOs, 33126400 bytes 2023-08-26T13:46:21.664 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:200 IOs, 106131456 bytes 2023-08-26T13:46:21.886 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:300 IOs, 164482048 bytes 2023-08-26T13:46:22.031 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:400 IOs, 188535808 bytes 2023-08-26T13:46:22.372 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:500 IOs, 300364800 bytes 2023-08-26T13:46:22.597 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:600 IOs, 372862976 bytes 2023-08-26T13:46:22.911 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:700 IOs, 461526016 bytes 2023-08-26T13:46:23.180 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:800 IOs, 543529984 bytes 2023-08-26T13:46:23.319 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:900 IOs, 560377856 bytes 2023-08-26T13:46:23.585 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:1000 IOs, 626921472 bytes 2023-08-26T13:46:23.805 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:1100 IOs, 692323328 bytes 2023-08-26T13:46:23.938 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:1200 IOs, 705758208 bytes 2023-08-26T13:46:24.156 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:1300 IOs, 770247680 bytes 2023-08-26T13:46:24.419 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:1400 IOs, 863273984 bytes 2023-08-26T13:46:24.646 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:1500 IOs, 882656256 bytes 2023-08-26T13:46:24.905 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:1600 IOs, 949610496 bytes 2023-08-26T13:46:25.150 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:1700 IOs, 1021955072 bytes 2023-08-26T13:46:25.366 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:1800 IOs, 1057056768 bytes 2023-08-26T13:46:25.504 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:1900 IOs, 1070471168 bytes 2023-08-26T13:46:25.726 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:2000 IOs, 1129473024 bytes 2023-08-26T13:46:25.927 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:2100 IOs, 1182627840 bytes 2023-08-26T13:46:26.114 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:2200 IOs, 1231512576 bytes 2023-08-26T13:46:26.301 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:2300 IOs, 1274988544 bytes 2023-08-26T13:46:26.447 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:2400 IOs, 1300534272 bytes 2023-08-26T13:46:26.603 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:2500 IOs, 1331464192 bytes 2023-08-26T13:46:26.774 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:2600 IOs, 1371767808 bytes 2023-08-26T13:46:26.957 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:2700 IOs, 1416115200 bytes 2023-08-26T13:46:27.211 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:2800 IOs, 1517637632 bytes 2023-08-26T13:46:27.371 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:2900 IOs, 1526318080 bytes 2023-08-26T13:46:27.514 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:3000 IOs, 1549585408 bytes 2023-08-26T13:46:27.714 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:3100 IOs, 1595559936 bytes 2023-08-26T13:46:27.811 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:3200 IOs, 1596165120 bytes 2023-08-26T13:46:28.048 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:3300 IOs, 1660716032 bytes 2023-08-26T13:46:28.211 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:3400 IOs, 1683414016 bytes 2023-08-26T13:46:28.384 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:3500 IOs, 1717899264 bytes 2023-08-26T13:46:28.535 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:3600 IOs, 1741736960 bytes 2023-08-26T13:46:28.744 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:3700 IOs, 1792112640 bytes 2023-08-26T13:46:28.853 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:3800 IOs, 1792721920 bytes 2023-08-26T13:46:29.010 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:3900 IOs, 1818454016 bytes 2023-08-26T13:46:29.151 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:4000 IOs, 1836381184 bytes 2023-08-26T13:46:29.257 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ error_code=124 2023-08-26T13:46:29.257 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ set -e 2023-08-26T13:46:29.257 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ '[' 124 -eq 124 ']' 2023-08-26T13:46:29.257 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ return 0 2023-08-26T13:46:29.258 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-08-26T13:46:29.258 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ image=image_24 2023-08-26T13:46:29.258 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ create_image_and_enable_mirror cluster2 mirror image_24 snapshot 128M 2023-08-26T13:46:29.258 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster2 2023-08-26T13:46:29.259 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ shift 2023-08-26T13:46:29.259 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T13:46:29.259 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ shift 2023-08-26T13:46:29.259 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=image_24 2023-08-26T13:46:29.259 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ shift 2023-08-26T13:46:29.260 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local mode=snapshot 2023-08-26T13:46:29.260 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ '[' -n snapshot ']' 2023-08-26T13:46:29.260 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ shift 2023-08-26T13:46:29.260 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ create_image cluster2 mirror image_24 128M 2023-08-26T13:46:29.260 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster2 2023-08-26T13:46:29.261 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ shift 2023-08-26T13:46:29.261 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T13:46:29.261 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ shift 2023-08-26T13:46:29.261 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=image_24 2023-08-26T13:46:29.262 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ shift 2023-08-26T13:46:29.262 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local size=128 2023-08-26T13:46:29.262 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ '[' -n 128M ']' 2023-08-26T13:46:29.262 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ size=128M 2023-08-26T13:46:29.262 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ shift 2023-08-26T13:46:29.263 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock mirror/image_24 2023-08-26T13:46:29.302 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ '[' image = image ']' 2023-08-26T13:46:29.302 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ enable_mirror cluster2 mirror image_24 snapshot 2023-08-26T13:46:29.303 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster2 2023-08-26T13:46:29.303 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T13:46:29.303 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=image_24 2023-08-26T13:46:29.303 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local mode=snapshot 2023-08-26T13:46:29.303 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_24 snapshot 2023-08-26T13:46:30.119 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:Mirroring enabled 2023-08-26T13:46:30.127 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rbd --cluster=cluster2 info mirror/image_24 2023-08-26T13:46:30.163 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:rbd image 'image_24': 2023-08-26T13:46:30.164 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: size 128 MiB in 32 objects 2023-08-26T13:46:30.164 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: order 22 (4 MiB objects) 2023-08-26T13:46:30.164 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: snapshot_count: 1 2023-08-26T13:46:30.164 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: id: 1365efb99ba6 2023-08-26T13:46:30.164 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: block_name_prefix: rbd_data.1365efb99ba6 2023-08-26T13:46:30.165 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: format: 2 2023-08-26T13:46:30.165 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: features: layering, exclusive-lock 2023-08-26T13:46:30.165 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: op_features: 2023-08-26T13:46:30.165 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: flags: 2023-08-26T13:46:30.165 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: create_timestamp: Sat Aug 26 13:46:29 2023 2023-08-26T13:46:30.166 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: access_timestamp: Sat Aug 26 13:46:29 2023 2023-08-26T13:46:30.166 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: modify_timestamp: Sat Aug 26 13:46:29 2023 2023-08-26T13:46:30.166 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: mirroring state: enabled 2023-08-26T13:46:30.166 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: mirroring mode: snapshot 2023-08-26T13:46:30.166 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: mirroring global id: bfc1dd79-696c-4322-bf38-f1dc8a31a580 2023-08-26T13:46:30.167 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: mirroring primary: true 2023-08-26T13:46:30.167 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ '[' -n '' ']' 2023-08-26T13:46:30.167 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ stress_write_image cluster2 mirror image_24 2023-08-26T13:46:30.168 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster2 2023-08-26T13:46:30.168 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T13:46:30.168 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=image_24 2023-08-26T13:46:30.168 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2023-08-26T13:46:30.169 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local duration=13 2023-08-26T13:46:30.169 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ set +e 2023-08-26T13:46:30.169 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ timeout 13s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_24 --debug-rbd=20 --debug-journaler=20 2023-08-26T13:46:30.387 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:100 IOs, 41880576 bytes 2023-08-26T13:46:30.532 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:200 IOs, 82529280 bytes 2023-08-26T13:46:30.662 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:300 IOs, 89453568 bytes 2023-08-26T13:46:30.848 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:400 IOs, 140072960 bytes 2023-08-26T13:46:31.069 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:500 IOs, 197508096 bytes 2023-08-26T13:46:31.251 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:600 IOs, 256872448 bytes 2023-08-26T13:46:31.500 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:700 IOs, 325621760 bytes 2023-08-26T13:46:31.687 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:800 IOs, 371483648 bytes 2023-08-26T13:46:31.849 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:900 IOs, 422557696 bytes 2023-08-26T13:46:32.028 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:1000 IOs, 456370176 bytes 2023-08-26T13:46:32.166 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:1100 IOs, 465489920 bytes 2023-08-26T13:46:32.328 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:1200 IOs, 489035776 bytes 2023-08-26T13:46:32.566 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:1300 IOs, 558466048 bytes 2023-08-26T13:46:32.692 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:1400 IOs, 576882688 bytes 2023-08-26T13:46:32.874 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:1500 IOs, 636364800 bytes 2023-08-26T13:46:33.003 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:1600 IOs, 636967936 bytes 2023-08-26T13:46:33.197 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:1700 IOs, 677285888 bytes 2023-08-26T13:46:33.404 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:1800 IOs, 729150464 bytes 2023-08-26T13:46:33.617 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:1900 IOs, 797035520 bytes 2023-08-26T13:46:33.843 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:2000 IOs, 856793088 bytes 2023-08-26T13:46:33.972 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:2100 IOs, 868928512 bytes 2023-08-26T13:46:34.084 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:2200 IOs, 886838272 bytes 2023-08-26T13:46:34.351 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:2300 IOs, 962229248 bytes 2023-08-26T13:46:34.481 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:2400 IOs, 984583168 bytes 2023-08-26T13:46:34.609 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:2500 IOs, 998675456 bytes 2023-08-26T13:46:34.731 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:2600 IOs, 1022248960 bytes 2023-08-26T13:46:35.010 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:2700 IOs, 1097916416 bytes 2023-08-26T13:46:35.191 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:2800 IOs, 1141331968 bytes 2023-08-26T13:46:35.428 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:2900 IOs, 1208972288 bytes 2023-08-26T13:46:35.592 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:3000 IOs, 1240427520 bytes 2023-08-26T13:46:35.865 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:3100 IOs, 1330008064 bytes 2023-08-26T13:46:36.009 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:3200 IOs, 1350772736 bytes 2023-08-26T13:46:36.221 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:3300 IOs, 1406241792 bytes 2023-08-26T13:46:36.364 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:3400 IOs, 1425163264 bytes 2023-08-26T13:46:36.560 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:3500 IOs, 1492105216 bytes 2023-08-26T13:46:36.696 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:3600 IOs, 1508230144 bytes 2023-08-26T13:46:36.926 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:3700 IOs, 1564333056 bytes 2023-08-26T13:46:37.137 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:3800 IOs, 1640024064 bytes 2023-08-26T13:46:37.277 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:3900 IOs, 1649037312 bytes 2023-08-26T13:46:37.576 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:4000 IOs, 1735469056 bytes 2023-08-26T13:46:37.688 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:4100 IOs, 1736097792 bytes 2023-08-26T13:46:37.880 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:4200 IOs, 1773283328 bytes 2023-08-26T13:46:38.088 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:4300 IOs, 1824932864 bytes 2023-08-26T13:46:38.319 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:4400 IOs, 1890271232 bytes 2023-08-26T13:46:38.544 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:4500 IOs, 1945541632 bytes 2023-08-26T13:46:38.817 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:4600 IOs, 2021604352 bytes 2023-08-26T13:46:38.965 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:4700 IOs, 2041688064 bytes 2023-08-26T13:46:39.102 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:4800 IOs, 2056236032 bytes 2023-08-26T13:46:39.213 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:4900 IOs, 2059581440 bytes 2023-08-26T13:46:39.424 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:5000 IOs, 2100646912 bytes 2023-08-26T13:46:39.570 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:5100 IOs, 2121123840 bytes 2023-08-26T13:46:39.854 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:5200 IOs, 2210449408 bytes 2023-08-26T13:46:40.095 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:5300 IOs, 2275189760 bytes 2023-08-26T13:46:40.236 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:5400 IOs, 2293914624 bytes 2023-08-26T13:46:40.491 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:5500 IOs, 2368545792 bytes 2023-08-26T13:46:40.633 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:5600 IOs, 2386253824 bytes 2023-08-26T13:46:40.831 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:5700 IOs, 2446224384 bytes 2023-08-26T13:46:41.022 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:5800 IOs, 2475960320 bytes 2023-08-26T13:46:41.142 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:5900 IOs, 2482673664 bytes 2023-08-26T13:46:41.347 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:6000 IOs, 2549606400 bytes 2023-08-26T13:46:41.566 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:6100 IOs, 2592354304 bytes 2023-08-26T13:46:41.731 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:6200 IOs, 2604630016 bytes 2023-08-26T13:46:41.972 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:6300 IOs, 2671667200 bytes 2023-08-26T13:46:42.214 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:6400 IOs, 2737088512 bytes 2023-08-26T13:46:42.404 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:6500 IOs, 2779830272 bytes 2023-08-26T13:46:42.616 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:6600 IOs, 2836117504 bytes 2023-08-26T13:46:42.790 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:6700 IOs, 2881562624 bytes 2023-08-26T13:46:43.003 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:6800 IOs, 2925195264 bytes 2023-08-26T13:46:43.175 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ error_code=124 2023-08-26T13:46:43.175 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ set -e 2023-08-26T13:46:43.175 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ '[' 124 -eq 124 ']' 2023-08-26T13:46:43.175 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ return 0 2023-08-26T13:46:43.176 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-08-26T13:46:43.176 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ image=image_25 2023-08-26T13:46:43.176 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ create_image_and_enable_mirror cluster2 mirror image_25 snapshot 128M 2023-08-26T13:46:43.176 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster2 2023-08-26T13:46:43.176 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ shift 2023-08-26T13:46:43.177 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T13:46:43.177 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ shift 2023-08-26T13:46:43.177 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=image_25 2023-08-26T13:46:43.177 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ shift 2023-08-26T13:46:43.177 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local mode=snapshot 2023-08-26T13:46:43.178 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ '[' -n snapshot ']' 2023-08-26T13:46:43.178 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ shift 2023-08-26T13:46:43.178 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ create_image cluster2 mirror image_25 128M 2023-08-26T13:46:43.178 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster2 2023-08-26T13:46:43.179 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ shift 2023-08-26T13:46:43.179 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T13:46:43.179 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ shift 2023-08-26T13:46:43.180 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=image_25 2023-08-26T13:46:43.180 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ shift 2023-08-26T13:46:43.180 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local size=128 2023-08-26T13:46:43.181 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ '[' -n 128M ']' 2023-08-26T13:46:43.181 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ size=128M 2023-08-26T13:46:43.181 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ shift 2023-08-26T13:46:43.181 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock mirror/image_25 2023-08-26T13:46:43.221 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ '[' image = image ']' 2023-08-26T13:46:43.280 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ enable_mirror cluster2 mirror image_25 snapshot 2023-08-26T13:46:43.280 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster2 2023-08-26T13:46:43.281 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T13:46:43.281 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=image_25 2023-08-26T13:46:43.281 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local mode=snapshot 2023-08-26T13:46:43.281 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_25 snapshot 2023-08-26T13:46:44.169 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:Mirroring enabled 2023-08-26T13:46:44.178 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rbd --cluster=cluster2 info mirror/image_25 2023-08-26T13:46:44.216 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:rbd image 'image_25': 2023-08-26T13:46:44.216 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: size 128 MiB in 32 objects 2023-08-26T13:46:44.216 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: order 22 (4 MiB objects) 2023-08-26T13:46:44.216 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: snapshot_count: 1 2023-08-26T13:46:44.216 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: id: 136fc9e7778b 2023-08-26T13:46:44.217 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: block_name_prefix: rbd_data.136fc9e7778b 2023-08-26T13:46:44.217 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: format: 2 2023-08-26T13:46:44.217 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: features: layering, exclusive-lock 2023-08-26T13:46:44.217 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: op_features: 2023-08-26T13:46:44.217 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: flags: 2023-08-26T13:46:44.218 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: create_timestamp: Sat Aug 26 13:46:43 2023 2023-08-26T13:46:44.218 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: access_timestamp: Sat Aug 26 13:46:43 2023 2023-08-26T13:46:44.218 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: modify_timestamp: Sat Aug 26 13:46:43 2023 2023-08-26T13:46:44.218 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: mirroring state: enabled 2023-08-26T13:46:44.218 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: mirroring mode: snapshot 2023-08-26T13:46:44.218 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: mirroring global id: ae7a1e38-d406-47df-97d4-9cbf52592bb7 2023-08-26T13:46:44.219 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: mirroring primary: true 2023-08-26T13:46:44.220 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ '[' -n '' ']' 2023-08-26T13:46:44.220 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ stress_write_image cluster2 mirror image_25 2023-08-26T13:46:44.220 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster2 2023-08-26T13:46:44.220 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T13:46:44.220 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=image_25 2023-08-26T13:46:44.221 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2023-08-26T13:46:44.221 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local duration=10 2023-08-26T13:46:44.221 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ set +e 2023-08-26T13:46:44.222 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ timeout 10s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_25 --debug-rbd=20 --debug-journaler=20 2023-08-26T13:46:44.417 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:100 IOs, 34361344 bytes 2023-08-26T13:46:44.530 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:200 IOs, 43523072 bytes 2023-08-26T13:46:44.678 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:300 IOs, 81859584 bytes 2023-08-26T13:46:44.855 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:400 IOs, 113343488 bytes 2023-08-26T13:46:45.004 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:500 IOs, 137606144 bytes 2023-08-26T13:46:45.187 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:600 IOs, 179046400 bytes 2023-08-26T13:46:45.338 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:700 IOs, 200633344 bytes 2023-08-26T13:46:45.461 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:800 IOs, 217465856 bytes 2023-08-26T13:46:45.705 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:900 IOs, 287099904 bytes 2023-08-26T13:46:45.857 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:1000 IOs, 302060544 bytes 2023-08-26T13:46:46.027 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:1100 IOs, 338862080 bytes 2023-08-26T13:46:46.131 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:1200 IOs, 339458048 bytes 2023-08-26T13:46:46.335 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:1300 IOs, 390209536 bytes 2023-08-26T13:46:46.566 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:1400 IOs, 457859072 bytes 2023-08-26T13:46:46.734 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:1500 IOs, 499904512 bytes 2023-08-26T13:46:46.879 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:1600 IOs, 522172416 bytes 2023-08-26T13:46:46.990 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:1700 IOs, 532143104 bytes 2023-08-26T13:46:47.131 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:1800 IOs, 569577472 bytes 2023-08-26T13:46:47.346 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:1900 IOs, 622848000 bytes 2023-08-26T13:46:47.564 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:2000 IOs, 684395520 bytes 2023-08-26T13:46:47.699 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:2100 IOs, 700223488 bytes 2023-08-26T13:46:47.804 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:2200 IOs, 700844032 bytes 2023-08-26T13:46:47.910 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:2300 IOs, 729113600 bytes 2023-08-26T13:46:48.073 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:2400 IOs, 741026816 bytes 2023-08-26T13:46:48.276 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:2500 IOs, 798833664 bytes 2023-08-26T13:46:48.432 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:2600 IOs, 826251264 bytes 2023-08-26T13:46:48.596 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:2700 IOs, 862437376 bytes 2023-08-26T13:46:48.784 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:2800 IOs, 893881344 bytes 2023-08-26T13:46:48.916 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:2900 IOs, 914244608 bytes 2023-08-26T13:46:49.072 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:3000 IOs, 938476544 bytes 2023-08-26T13:46:49.255 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:3100 IOs, 980714496 bytes 2023-08-26T13:46:49.429 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:3200 IOs, 1022826496 bytes 2023-08-26T13:46:49.630 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:3300 IOs, 1076725760 bytes 2023-08-26T13:46:49.765 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:3400 IOs, 1086612480 bytes 2023-08-26T13:46:50.089 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:3500 IOs, 1196461056 bytes 2023-08-26T13:46:50.303 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:3600 IOs, 1247249408 bytes 2023-08-26T13:46:50.503 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:3700 IOs, 1303895040 bytes 2023-08-26T13:46:50.617 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:3800 IOs, 1337670656 bytes 2023-08-26T13:46:50.774 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:3900 IOs, 1338276864 bytes 2023-08-26T13:46:50.962 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:4000 IOs, 1386504192 bytes 2023-08-26T13:46:51.097 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:4100 IOs, 1402783744 bytes 2023-08-26T13:46:51.272 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:4200 IOs, 1442174976 bytes 2023-08-26T13:46:51.379 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:4300 IOs, 1449392128 bytes 2023-08-26T13:46:51.527 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:4400 IOs, 1471299584 bytes 2023-08-26T13:46:51.764 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:4500 IOs, 1543783424 bytes 2023-08-26T13:46:51.867 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:4600 IOs, 1548782592 bytes 2023-08-26T13:46:52.020 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:4700 IOs, 1573801984 bytes 2023-08-26T13:46:52.185 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:4800 IOs, 1599775744 bytes 2023-08-26T13:46:52.369 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:4900 IOs, 1639116800 bytes 2023-08-26T13:46:52.474 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:5000 IOs, 1639706624 bytes 2023-08-26T13:46:52.633 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:5100 IOs, 1665099776 bytes 2023-08-26T13:46:52.921 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:5200 IOs, 1691853824 bytes 2023-08-26T13:46:53.046 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:5300 IOs, 1704602624 bytes 2023-08-26T13:46:53.314 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:5400 IOs, 1786418176 bytes 2023-08-26T13:46:53.486 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:5500 IOs, 1817511936 bytes 2023-08-26T13:46:53.765 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:5600 IOs, 1908234240 bytes 2023-08-26T13:46:53.930 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:5700 IOs, 1937682432 bytes 2023-08-26T13:46:54.081 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:5800 IOs, 1967128576 bytes 2023-08-26T13:46:54.226 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ error_code=124 2023-08-26T13:46:54.227 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ set -e 2023-08-26T13:46:54.227 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ '[' 124 -eq 124 ']' 2023-08-26T13:46:54.227 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ return 0 2023-08-26T13:46:54.227 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-08-26T13:46:54.227 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ image=image_26 2023-08-26T13:46:54.228 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ create_image_and_enable_mirror cluster2 mirror image_26 snapshot 128M 2023-08-26T13:46:54.228 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster2 2023-08-26T13:46:54.228 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ shift 2023-08-26T13:46:54.228 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T13:46:54.228 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ shift 2023-08-26T13:46:54.229 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=image_26 2023-08-26T13:46:54.229 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ shift 2023-08-26T13:46:54.229 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local mode=snapshot 2023-08-26T13:46:54.229 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ '[' -n snapshot ']' 2023-08-26T13:46:54.229 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ shift 2023-08-26T13:46:54.230 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ create_image cluster2 mirror image_26 128M 2023-08-26T13:46:54.230 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster2 2023-08-26T13:46:54.230 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ shift 2023-08-26T13:46:54.231 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T13:46:54.231 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ shift 2023-08-26T13:46:54.231 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=image_26 2023-08-26T13:46:54.231 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ shift 2023-08-26T13:46:54.232 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local size=128 2023-08-26T13:46:54.232 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ '[' -n 128M ']' 2023-08-26T13:46:54.232 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ size=128M 2023-08-26T13:46:54.232 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ shift 2023-08-26T13:46:54.232 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock mirror/image_26 2023-08-26T13:46:54.272 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ '[' image = image ']' 2023-08-26T13:46:54.273 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ enable_mirror cluster2 mirror image_26 snapshot 2023-08-26T13:46:54.273 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster2 2023-08-26T13:46:54.273 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T13:46:54.273 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=image_26 2023-08-26T13:46:54.274 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local mode=snapshot 2023-08-26T13:46:54.274 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_26 snapshot 2023-08-26T13:46:54.962 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:Mirroring enabled 2023-08-26T13:46:54.970 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rbd --cluster=cluster2 info mirror/image_26 2023-08-26T13:46:55.008 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:rbd image 'image_26': 2023-08-26T13:46:55.009 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: size 128 MiB in 32 objects 2023-08-26T13:46:55.009 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: order 22 (4 MiB objects) 2023-08-26T13:46:55.009 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: snapshot_count: 1 2023-08-26T13:46:55.009 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: id: 1379ac0c3b13 2023-08-26T13:46:55.010 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: block_name_prefix: rbd_data.1379ac0c3b13 2023-08-26T13:46:55.010 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: format: 2 2023-08-26T13:46:55.010 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: features: layering, exclusive-lock 2023-08-26T13:46:55.010 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: op_features: 2023-08-26T13:46:55.010 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: flags: 2023-08-26T13:46:55.011 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: create_timestamp: Sat Aug 26 13:46:54 2023 2023-08-26T13:46:55.011 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: access_timestamp: Sat Aug 26 13:46:54 2023 2023-08-26T13:46:55.011 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: modify_timestamp: Sat Aug 26 13:46:54 2023 2023-08-26T13:46:55.011 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: mirroring state: enabled 2023-08-26T13:46:55.011 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: mirroring mode: snapshot 2023-08-26T13:46:55.012 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: mirroring global id: ef7edecb-ca31-4518-87d6-58c3dfe520f7 2023-08-26T13:46:55.012 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: mirroring primary: true 2023-08-26T13:46:55.013 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ '[' -n '' ']' 2023-08-26T13:46:55.013 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ stress_write_image cluster2 mirror image_26 2023-08-26T13:46:55.013 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster2 2023-08-26T13:46:55.013 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T13:46:55.013 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=image_26 2023-08-26T13:46:55.014 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2023-08-26T13:46:55.014 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local duration=5 2023-08-26T13:46:55.014 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ set +e 2023-08-26T13:46:55.015 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ timeout 5s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_26 --debug-rbd=20 --debug-journaler=20 2023-08-26T13:46:55.287 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:100 IOs, 56801280 bytes 2023-08-26T13:46:55.469 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:200 IOs, 100584448 bytes 2023-08-26T13:46:55.601 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:300 IOs, 116438016 bytes 2023-08-26T13:46:55.760 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:400 IOs, 141730816 bytes 2023-08-26T13:46:55.971 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:500 IOs, 203483136 bytes 2023-08-26T13:46:56.161 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:600 IOs, 240346112 bytes 2023-08-26T13:46:56.403 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:700 IOs, 324688896 bytes 2023-08-26T13:46:56.642 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:800 IOs, 369568768 bytes 2023-08-26T13:46:56.832 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:900 IOs, 413524992 bytes 2023-08-26T13:46:57.030 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:1000 IOs, 479993856 bytes 2023-08-26T13:46:57.214 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:1100 IOs, 503517184 bytes 2023-08-26T13:46:57.342 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:1200 IOs, 516130816 bytes 2023-08-26T13:46:57.552 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:1300 IOs, 571195392 bytes 2023-08-26T13:46:57.797 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:1400 IOs, 648904704 bytes 2023-08-26T13:46:57.962 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:1500 IOs, 674694144 bytes 2023-08-26T13:46:58.116 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:1600 IOs, 693583872 bytes 2023-08-26T13:46:58.345 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:1700 IOs, 758638592 bytes 2023-08-26T13:46:58.451 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:1800 IOs, 771727360 bytes 2023-08-26T13:46:58.667 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:1900 IOs, 822952960 bytes 2023-08-26T13:46:58.809 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:2000 IOs, 842275840 bytes 2023-08-26T13:46:58.935 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:2100 IOs, 855614464 bytes 2023-08-26T13:46:59.129 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:2200 IOs, 905719808 bytes 2023-08-26T13:46:59.301 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:2300 IOs, 948674560 bytes 2023-08-26T13:46:59.471 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:2400 IOs, 976404480 bytes 2023-08-26T13:46:59.630 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:2500 IOs, 1012783104 bytes 2023-08-26T13:46:59.760 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:2600 IOs, 1013412864 bytes 2023-08-26T13:46:59.877 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:2700 IOs, 1020229632 bytes 2023-08-26T13:47:00.020 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ error_code=124 2023-08-26T13:47:00.021 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ set -e 2023-08-26T13:47:00.021 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ '[' 124 -eq 124 ']' 2023-08-26T13:47:00.021 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ return 0 2023-08-26T13:47:00.021 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-08-26T13:47:00.021 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ image=image_27 2023-08-26T13:47:00.022 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ create_image_and_enable_mirror cluster2 mirror image_27 snapshot 128M 2023-08-26T13:47:00.022 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster2 2023-08-26T13:47:00.022 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ shift 2023-08-26T13:47:00.022 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T13:47:00.022 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ shift 2023-08-26T13:47:00.023 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=image_27 2023-08-26T13:47:00.023 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ shift 2023-08-26T13:47:00.023 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local mode=snapshot 2023-08-26T13:47:00.023 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ '[' -n snapshot ']' 2023-08-26T13:47:00.023 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ shift 2023-08-26T13:47:00.023 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ create_image cluster2 mirror image_27 128M 2023-08-26T13:47:00.024 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster2 2023-08-26T13:47:00.024 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ shift 2023-08-26T13:47:00.024 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T13:47:00.024 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ shift 2023-08-26T13:47:00.024 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=image_27 2023-08-26T13:47:00.025 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ shift 2023-08-26T13:47:00.025 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local size=128 2023-08-26T13:47:00.025 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ '[' -n 128M ']' 2023-08-26T13:47:00.025 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ size=128M 2023-08-26T13:47:00.025 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ shift 2023-08-26T13:47:00.026 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock mirror/image_27 2023-08-26T13:47:00.066 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ '[' image = image ']' 2023-08-26T13:47:00.067 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ enable_mirror cluster2 mirror image_27 snapshot 2023-08-26T13:47:00.067 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster2 2023-08-26T13:47:00.067 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T13:47:00.068 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=image_27 2023-08-26T13:47:00.068 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local mode=snapshot 2023-08-26T13:47:00.068 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_27 snapshot 2023-08-26T13:47:01.129 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:Mirroring enabled 2023-08-26T13:47:01.138 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rbd --cluster=cluster2 info mirror/image_27 2023-08-26T13:47:01.178 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:rbd image 'image_27': 2023-08-26T13:47:01.179 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: size 128 MiB in 32 objects 2023-08-26T13:47:01.179 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: order 22 (4 MiB objects) 2023-08-26T13:47:01.179 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: snapshot_count: 1 2023-08-26T13:47:01.179 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: id: 138328998392 2023-08-26T13:47:01.179 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: block_name_prefix: rbd_data.138328998392 2023-08-26T13:47:01.180 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: format: 2 2023-08-26T13:47:01.180 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: features: layering, exclusive-lock 2023-08-26T13:47:01.180 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: op_features: 2023-08-26T13:47:01.180 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: flags: 2023-08-26T13:47:01.180 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: create_timestamp: Sat Aug 26 13:47:00 2023 2023-08-26T13:47:01.181 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: access_timestamp: Sat Aug 26 13:47:00 2023 2023-08-26T13:47:01.181 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: modify_timestamp: Sat Aug 26 13:47:00 2023 2023-08-26T13:47:01.181 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: mirroring state: enabled 2023-08-26T13:47:01.181 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: mirroring mode: snapshot 2023-08-26T13:47:01.181 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: mirroring global id: bceda8a0-e824-4b77-b33c-1ee70e5c9a49 2023-08-26T13:47:01.182 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: mirroring primary: true 2023-08-26T13:47:01.182 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ '[' -n '' ']' 2023-08-26T13:47:01.182 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ stress_write_image cluster2 mirror image_27 2023-08-26T13:47:01.183 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster2 2023-08-26T13:47:01.183 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T13:47:01.183 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=image_27 2023-08-26T13:47:01.183 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2023-08-26T13:47:01.184 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local duration=7 2023-08-26T13:47:01.184 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ set +e 2023-08-26T13:47:01.184 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ timeout 7s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_27 --debug-rbd=20 --debug-journaler=20 2023-08-26T13:47:01.448 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:100 IOs, 65823744 bytes 2023-08-26T13:47:01.605 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:200 IOs, 84720640 bytes 2023-08-26T13:47:01.827 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:300 IOs, 147708928 bytes 2023-08-26T13:47:01.982 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:400 IOs, 170858496 bytes 2023-08-26T13:47:02.187 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:500 IOs, 226203648 bytes 2023-08-26T13:47:02.351 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:600 IOs, 262620160 bytes 2023-08-26T13:47:02.507 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:700 IOs, 293570560 bytes 2023-08-26T13:47:02.676 INFO:tasks.ceph.cluster2.osd.2.smithi124.stderr:2023-08-26T13:47:02.675+0000 7fb790854700 -1 reset not still connected to 0x5598e2645450 2023-08-26T13:47:02.688 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:800 IOs, 328345600 bytes 2023-08-26T13:47:02.843 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:900 IOs, 358203392 bytes 2023-08-26T13:47:02.974 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:1000 IOs, 376803328 bytes 2023-08-26T13:47:03.099 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:1100 IOs, 387612672 bytes 2023-08-26T13:47:03.233 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:1200 IOs, 402223104 bytes 2023-08-26T13:47:03.375 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:1300 IOs, 424955904 bytes 2023-08-26T13:47:03.631 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:1400 IOs, 497513472 bytes 2023-08-26T13:47:03.817 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:1500 IOs, 541227008 bytes 2023-08-26T13:47:03.959 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:1600 IOs, 563395584 bytes 2023-08-26T13:47:04.104 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:1700 IOs, 582180864 bytes 2023-08-26T13:47:04.274 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:1800 IOs, 614729728 bytes 2023-08-26T13:47:04.423 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:1900 IOs, 639232000 bytes 2023-08-26T13:47:04.596 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:2000 IOs, 677918720 bytes 2023-08-26T13:47:04.767 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:2100 IOs, 711373824 bytes 2023-08-26T13:47:04.968 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:2200 IOs, 762125312 bytes 2023-08-26T13:47:05.132 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:2300 IOs, 796616704 bytes 2023-08-26T13:47:05.284 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:2400 IOs, 822126592 bytes 2023-08-26T13:47:05.471 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:2500 IOs, 869172224 bytes 2023-08-26T13:47:05.682 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:2600 IOs, 922530816 bytes 2023-08-26T13:47:05.931 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:2700 IOs, 992644096 bytes 2023-08-26T13:47:06.123 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:2800 IOs, 1031808000 bytes 2023-08-26T13:47:06.368 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:2900 IOs, 1102542848 bytes 2023-08-26T13:47:06.541 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:3000 IOs, 1138120704 bytes 2023-08-26T13:47:06.707 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:3100 IOs, 1172481024 bytes 2023-08-26T13:47:06.849 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:3200 IOs, 1204463616 bytes 2023-08-26T13:47:07.070 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:3300 IOs, 1253197824 bytes 2023-08-26T13:47:07.341 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:3400 IOs, 1339209728 bytes 2023-08-26T13:47:07.584 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:3500 IOs, 1401397248 bytes 2023-08-26T13:47:07.731 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:3600 IOs, 1423032320 bytes 2023-08-26T13:47:07.952 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:3700 IOs, 1488304128 bytes 2023-08-26T13:47:08.067 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:3800 IOs, 1488891904 bytes 2023-08-26T13:47:08.192 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ error_code=124 2023-08-26T13:47:08.193 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ set -e 2023-08-26T13:47:08.193 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ '[' 124 -eq 124 ']' 2023-08-26T13:47:08.193 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ return 0 2023-08-26T13:47:08.193 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-08-26T13:47:08.194 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ image=image_28 2023-08-26T13:47:08.194 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ create_image_and_enable_mirror cluster2 mirror image_28 snapshot 128M 2023-08-26T13:47:08.194 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster2 2023-08-26T13:47:08.194 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ shift 2023-08-26T13:47:08.194 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T13:47:08.195 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ shift 2023-08-26T13:47:08.195 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=image_28 2023-08-26T13:47:08.195 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ shift 2023-08-26T13:47:08.195 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local mode=snapshot 2023-08-26T13:47:08.195 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ '[' -n snapshot ']' 2023-08-26T13:47:08.196 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ shift 2023-08-26T13:47:08.196 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ create_image cluster2 mirror image_28 128M 2023-08-26T13:47:08.196 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster2 2023-08-26T13:47:08.196 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ shift 2023-08-26T13:47:08.196 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T13:47:08.196 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ shift 2023-08-26T13:47:08.197 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=image_28 2023-08-26T13:47:08.197 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ shift 2023-08-26T13:47:08.197 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local size=128 2023-08-26T13:47:08.197 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ '[' -n 128M ']' 2023-08-26T13:47:08.198 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ size=128M 2023-08-26T13:47:08.198 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ shift 2023-08-26T13:47:08.198 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock mirror/image_28 2023-08-26T13:47:08.240 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ '[' image = image ']' 2023-08-26T13:47:08.240 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ enable_mirror cluster2 mirror image_28 snapshot 2023-08-26T13:47:08.240 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster2 2023-08-26T13:47:08.240 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T13:47:08.240 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=image_28 2023-08-26T13:47:08.241 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local mode=snapshot 2023-08-26T13:47:08.241 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_28 snapshot 2023-08-26T13:47:09.121 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:Mirroring enabled 2023-08-26T13:47:09.130 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rbd --cluster=cluster2 info mirror/image_28 2023-08-26T13:47:09.170 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:rbd image 'image_28': 2023-08-26T13:47:09.170 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: size 128 MiB in 32 objects 2023-08-26T13:47:09.170 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: order 22 (4 MiB objects) 2023-08-26T13:47:09.170 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: snapshot_count: 1 2023-08-26T13:47:09.171 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: id: 138c37a313f9 2023-08-26T13:47:09.171 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: block_name_prefix: rbd_data.138c37a313f9 2023-08-26T13:47:09.171 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: format: 2 2023-08-26T13:47:09.171 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: features: layering, exclusive-lock 2023-08-26T13:47:09.171 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: op_features: 2023-08-26T13:47:09.172 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: flags: 2023-08-26T13:47:09.172 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: create_timestamp: Sat Aug 26 13:47:08 2023 2023-08-26T13:47:09.172 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: access_timestamp: Sat Aug 26 13:47:08 2023 2023-08-26T13:47:09.172 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: modify_timestamp: Sat Aug 26 13:47:08 2023 2023-08-26T13:47:09.172 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: mirroring state: enabled 2023-08-26T13:47:09.172 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: mirroring mode: snapshot 2023-08-26T13:47:09.173 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: mirroring global id: b66f96d3-c7f1-4362-870d-f32a84410a1f 2023-08-26T13:47:09.173 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: mirroring primary: true 2023-08-26T13:47:09.174 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ '[' -n '' ']' 2023-08-26T13:47:09.174 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ stress_write_image cluster2 mirror image_28 2023-08-26T13:47:09.174 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster2 2023-08-26T13:47:09.174 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T13:47:09.175 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=image_28 2023-08-26T13:47:09.175 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2023-08-26T13:47:09.175 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local duration=14 2023-08-26T13:47:09.176 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ set +e 2023-08-26T13:47:09.176 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ timeout 14s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_28 --debug-rbd=20 --debug-journaler=20 2023-08-26T13:47:09.358 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:100 IOs, 18651136 bytes 2023-08-26T13:47:09.489 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:200 IOs, 31036416 bytes 2023-08-26T13:47:09.685 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:300 IOs, 85404672 bytes 2023-08-26T13:47:09.881 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:400 IOs, 134079488 bytes 2023-08-26T13:47:10.150 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:500 IOs, 220108800 bytes 2023-08-26T13:47:10.339 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:600 IOs, 266670080 bytes 2023-08-26T13:47:10.507 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:700 IOs, 311890944 bytes 2023-08-26T13:47:10.761 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:800 IOs, 386258944 bytes 2023-08-26T13:47:10.939 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:900 IOs, 429329408 bytes 2023-08-26T13:47:11.083 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:1000 IOs, 451582976 bytes 2023-08-26T13:47:11.193 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:1100 IOs, 452198400 bytes 2023-08-26T13:47:11.299 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:1200 IOs, 454281216 bytes 2023-08-26T13:47:11.486 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:1300 IOs, 498368512 bytes 2023-08-26T13:47:11.735 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:1400 IOs, 570031104 bytes 2023-08-26T13:47:11.839 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:1500 IOs, 570678272 bytes 2023-08-26T13:47:12.017 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:1600 IOs, 622788608 bytes 2023-08-26T13:47:12.203 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:1700 IOs, 646985728 bytes 2023-08-26T13:47:12.436 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:1800 IOs, 720223232 bytes 2023-08-26T13:47:12.656 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:1900 IOs, 774245376 bytes 2023-08-26T13:47:12.875 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:2000 IOs, 840468480 bytes 2023-08-26T13:47:13.112 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:2100 IOs, 906349568 bytes 2023-08-26T13:47:13.406 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:2200 IOs, 1006280704 bytes 2023-08-26T13:47:13.555 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:2300 IOs, 1037579264 bytes 2023-08-26T13:47:13.684 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:2400 IOs, 1048321024 bytes 2023-08-26T13:47:13.872 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:2500 IOs, 1092917248 bytes 2023-08-26T13:47:14.014 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:2600 IOs, 1114301440 bytes 2023-08-26T13:47:14.172 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:2700 IOs, 1145145344 bytes 2023-08-26T13:47:14.277 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:2800 IOs, 1145773056 bytes 2023-08-26T13:47:14.495 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:2900 IOs, 1205232640 bytes 2023-08-26T13:47:14.622 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:3000 IOs, 1218956288 bytes 2023-08-26T13:47:14.835 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:3100 IOs, 1273064448 bytes 2023-08-26T13:47:14.963 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:3200 IOs, 1289762816 bytes 2023-08-26T13:47:15.218 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:3300 IOs, 1367044096 bytes 2023-08-26T13:47:15.364 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:3400 IOs, 1389248512 bytes 2023-08-26T13:47:15.499 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:3500 IOs, 1406958592 bytes 2023-08-26T13:47:15.720 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:3600 IOs, 1479701504 bytes 2023-08-26T13:47:15.897 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:3700 IOs, 1504048128 bytes 2023-08-26T13:47:15.997 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:3800 IOs, 1526976512 bytes 2023-08-26T13:47:16.199 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:3900 IOs, 1566231552 bytes 2023-08-26T13:47:16.370 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:4000 IOs, 1585576960 bytes 2023-08-26T13:47:16.661 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:4100 IOs, 1684433920 bytes 2023-08-26T13:47:16.819 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:4200 IOs, 1714019328 bytes 2023-08-26T13:47:17.002 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:4300 IOs, 1759028224 bytes 2023-08-26T13:47:17.212 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:4400 IOs, 1812687872 bytes 2023-08-26T13:47:17.365 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:4500 IOs, 1861414912 bytes 2023-08-26T13:47:17.513 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:4600 IOs, 1866773504 bytes 2023-08-26T13:47:17.661 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:4700 IOs, 1881024512 bytes 2023-08-26T13:47:17.804 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:4800 IOs, 1905302528 bytes 2023-08-26T13:47:17.960 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:4900 IOs, 1948374016 bytes 2023-08-26T13:47:18.159 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:5000 IOs, 1987431424 bytes 2023-08-26T13:47:18.346 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:5100 IOs, 2029151232 bytes 2023-08-26T13:47:18.530 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:5200 IOs, 2067713024 bytes 2023-08-26T13:47:18.636 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:5300 IOs, 2068332544 bytes 2023-08-26T13:47:18.752 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:5400 IOs, 2081417216 bytes 2023-08-26T13:47:18.969 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:5500 IOs, 2143856640 bytes 2023-08-26T13:47:19.243 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:5600 IOs, 2253947904 bytes 2023-08-26T13:47:19.400 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:5700 IOs, 2265327616 bytes 2023-08-26T13:47:19.596 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:5800 IOs, 2308521984 bytes 2023-08-26T13:47:19.758 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:5900 IOs, 2338553856 bytes 2023-08-26T13:47:19.885 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:6000 IOs, 2351826944 bytes 2023-08-26T13:47:20.123 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:6100 IOs, 2423885824 bytes 2023-08-26T13:47:20.322 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:6200 IOs, 2470924288 bytes 2023-08-26T13:47:20.458 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:6300 IOs, 2488200192 bytes 2023-08-26T13:47:20.702 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:6400 IOs, 2562098176 bytes 2023-08-26T13:47:20.716 INFO:tasks.rbd_mirror_thrash:kill cluster1.client.mirror.0 2023-08-26T13:47:20.716 INFO:tasks.rbd_mirror.cluster1.client.mirror.0:Sent signal 15 2023-08-26T13:47:20.716 INFO:tasks.rbd_mirror_thrash:waiting for 4 secs before reviving daemons 2023-08-26T13:47:20.717 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi124.stderr:2023-08-26T13:47:20.715+0000 7f7e67a6f700 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.0 (PID: 115702) UID: 1000 2023-08-26T13:47:20.910 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:6500 IOs, 2614847488 bytes 2023-08-26T13:47:21.036 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:6600 IOs, 2624988160 bytes 2023-08-26T13:47:21.156 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:6700 IOs, 2650626048 bytes 2023-08-26T13:47:21.348 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:6800 IOs, 2679209984 bytes 2023-08-26T13:47:21.573 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:6900 IOs, 2746244096 bytes 2023-08-26T13:47:21.696 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:7000 IOs, 2758876160 bytes 2023-08-26T13:47:21.860 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:7100 IOs, 2788948992 bytes 2023-08-26T13:47:22.004 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:7200 IOs, 2814629888 bytes 2023-08-26T13:47:22.233 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:7300 IOs, 2866728960 bytes 2023-08-26T13:47:22.415 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:7400 IOs, 2907951104 bytes 2023-08-26T13:47:22.548 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:7500 IOs, 2939120640 bytes 2023-08-26T13:47:22.703 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:7600 IOs, 2946792448 bytes 2023-08-26T13:47:22.973 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:7700 IOs, 3009157120 bytes 2023-08-26T13:47:23.149 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:7800 IOs, 3045288960 bytes 2023-08-26T13:47:23.192 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ error_code=124 2023-08-26T13:47:23.192 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ set -e 2023-08-26T13:47:23.193 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ '[' 124 -eq 124 ']' 2023-08-26T13:47:23.193 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ return 0 2023-08-26T13:47:23.193 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-08-26T13:47:23.193 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ image=image_29 2023-08-26T13:47:23.193 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ create_image_and_enable_mirror cluster2 mirror image_29 snapshot 128M 2023-08-26T13:47:23.194 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster2 2023-08-26T13:47:23.194 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ shift 2023-08-26T13:47:23.194 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T13:47:23.194 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ shift 2023-08-26T13:47:23.195 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=image_29 2023-08-26T13:47:23.195 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ shift 2023-08-26T13:47:23.195 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local mode=snapshot 2023-08-26T13:47:23.196 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ '[' -n snapshot ']' 2023-08-26T13:47:23.196 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ shift 2023-08-26T13:47:23.196 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ create_image cluster2 mirror image_29 128M 2023-08-26T13:47:23.196 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster2 2023-08-26T13:47:23.196 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ shift 2023-08-26T13:47:23.196 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T13:47:23.197 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ shift 2023-08-26T13:47:23.197 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=image_29 2023-08-26T13:47:23.197 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ shift 2023-08-26T13:47:23.197 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local size=128 2023-08-26T13:47:23.197 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ '[' -n 128M ']' 2023-08-26T13:47:23.198 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ size=128M 2023-08-26T13:47:23.198 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ shift 2023-08-26T13:47:23.198 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock mirror/image_29 2023-08-26T13:47:23.243 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ '[' image = image ']' 2023-08-26T13:47:23.243 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ enable_mirror cluster2 mirror image_29 snapshot 2023-08-26T13:47:23.243 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster2 2023-08-26T13:47:23.243 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T13:47:23.244 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=image_29 2023-08-26T13:47:23.244 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local mode=snapshot 2023-08-26T13:47:23.244 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_29 snapshot 2023-08-26T13:47:24.194 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:Mirroring enabled 2023-08-26T13:47:24.203 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rbd --cluster=cluster2 info mirror/image_29 2023-08-26T13:47:24.241 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:rbd image 'image_29': 2023-08-26T13:47:24.241 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: size 128 MiB in 32 objects 2023-08-26T13:47:24.241 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: order 22 (4 MiB objects) 2023-08-26T13:47:24.241 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: snapshot_count: 1 2023-08-26T13:47:24.242 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: id: 1394424db2c2 2023-08-26T13:47:24.242 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: block_name_prefix: rbd_data.1394424db2c2 2023-08-26T13:47:24.242 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: format: 2 2023-08-26T13:47:24.242 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: features: layering, exclusive-lock 2023-08-26T13:47:24.242 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: op_features: 2023-08-26T13:47:24.243 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: flags: 2023-08-26T13:47:24.243 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: create_timestamp: Sat Aug 26 13:47:23 2023 2023-08-26T13:47:24.243 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: access_timestamp: Sat Aug 26 13:47:23 2023 2023-08-26T13:47:24.243 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: modify_timestamp: Sat Aug 26 13:47:23 2023 2023-08-26T13:47:24.243 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: mirroring state: enabled 2023-08-26T13:47:24.244 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: mirroring mode: snapshot 2023-08-26T13:47:24.244 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: mirroring global id: 2286d3a2-d0dc-442e-9a23-9c83c4633272 2023-08-26T13:47:24.244 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: mirroring primary: true 2023-08-26T13:47:24.245 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ '[' -n '' ']' 2023-08-26T13:47:24.245 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ stress_write_image cluster2 mirror image_29 2023-08-26T13:47:24.245 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster2 2023-08-26T13:47:24.246 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T13:47:24.246 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=image_29 2023-08-26T13:47:24.246 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2023-08-26T13:47:24.247 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local duration=13 2023-08-26T13:47:24.247 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ set +e 2023-08-26T13:47:24.247 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ timeout 13s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_29 --debug-rbd=20 --debug-journaler=20 2023-08-26T13:47:24.433 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:100 IOs, 21827584 bytes 2023-08-26T13:47:24.572 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:200 IOs, 52957184 bytes 2023-08-26T13:47:24.699 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:300 IOs, 53577728 bytes 2023-08-26T13:47:24.717 INFO:tasks.rbd_mirror_thrash:waiting for cluster1.client.mirror.0 2023-08-26T13:47:24.718 INFO:teuthology.orchestra.run:waiting for 600 2023-08-26T13:47:24.718 INFO:tasks.rbd_mirror_thrash:reviving cluster1.client.mirror.0 2023-08-26T13:47:24.718 INFO:tasks.rbd_mirror.cluster1.client.mirror.0:Restarting daemon 2023-08-26T13:47:24.718 DEBUG:teuthology.orchestra.run.smithi124:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.0 2023-08-26T13:47:24.721 INFO:tasks.rbd_mirror.cluster1.client.mirror.0:Started 2023-08-26T13:47:24.721 INFO:tasks.rbd_mirror_thrash:waiting for 119 secs before thrashing 2023-08-26T13:47:25.006 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:400 IOs, 113570816 bytes 2023-08-26T13:47:25.120 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:500 IOs, 114185216 bytes 2023-08-26T13:47:25.376 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:600 IOs, 199995392 bytes 2023-08-26T13:47:25.567 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:700 IOs, 245173248 bytes 2023-08-26T13:47:25.734 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:800 IOs, 278828032 bytes 2023-08-26T13:47:25.835 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:900 IOs, 279440384 bytes 2023-08-26T13:47:25.974 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:1000 IOs, 301591552 bytes 2023-08-26T13:47:26.160 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:1100 IOs, 344345600 bytes 2023-08-26T13:47:26.326 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:1200 IOs, 373177344 bytes 2023-08-26T13:47:26.428 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:1300 IOs, 373805056 bytes 2023-08-26T13:47:26.607 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:1400 IOs, 421469184 bytes 2023-08-26T13:47:26.766 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:1500 IOs, 453599232 bytes 2023-08-26T13:47:27.005 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:1600 IOs, 524650496 bytes 2023-08-26T13:47:27.153 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:1700 IOs, 554119168 bytes 2023-08-26T13:47:27.340 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:1800 IOs, 598252544 bytes 2023-08-26T13:47:27.445 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:1900 IOs, 598860800 bytes 2023-08-26T13:47:27.557 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:2000 IOs, 605240320 bytes 2023-08-26T13:47:27.803 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:2100 IOs, 660051968 bytes 2023-08-26T13:47:27.982 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:2200 IOs, 702363648 bytes 2023-08-26T13:47:28.183 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:2300 IOs, 753403904 bytes 2023-08-26T13:47:28.446 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:2400 IOs, 832460800 bytes 2023-08-26T13:47:28.603 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:2500 IOs, 858037248 bytes 2023-08-26T13:47:28.815 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:2600 IOs, 920750080 bytes 2023-08-26T13:47:28.949 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:2700 IOs, 943351808 bytes 2023-08-26T13:47:29.070 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:2800 IOs, 943972352 bytes 2023-08-26T13:47:29.209 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:2900 IOs, 964092928 bytes 2023-08-26T13:47:29.395 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:3000 IOs, 1009291264 bytes 2023-08-26T13:47:29.594 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:3100 IOs, 1058540544 bytes 2023-08-26T13:47:29.757 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:3200 IOs, 1090782208 bytes 2023-08-26T13:47:29.945 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:3300 IOs, 1132492800 bytes 2023-08-26T13:47:30.050 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:3400 IOs, 1133109248 bytes 2023-08-26T13:47:30.265 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:3500 IOs, 1192903680 bytes 2023-08-26T13:47:30.445 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:3600 IOs, 1235294208 bytes 2023-08-26T13:47:30.600 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:3700 IOs, 1261876224 bytes 2023-08-26T13:47:30.747 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:3800 IOs, 1285622784 bytes 2023-08-26T13:47:30.908 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:3900 IOs, 1342920704 bytes 2023-08-26T13:47:31.136 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:4000 IOs, 1380601856 bytes 2023-08-26T13:47:31.294 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:4100 IOs, 1412300800 bytes 2023-08-26T13:47:31.473 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:4200 IOs, 1456874496 bytes 2023-08-26T13:47:31.625 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:4300 IOs, 1485870080 bytes 2023-08-26T13:47:31.808 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:4400 IOs, 1530627072 bytes 2023-08-26T13:47:32.008 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:4500 IOs, 1576832000 bytes 2023-08-26T13:47:32.222 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:4600 IOs, 1634599936 bytes 2023-08-26T13:47:32.380 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:4700 IOs, 1658194944 bytes 2023-08-26T13:47:32.594 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:4800 IOs, 1722118144 bytes 2023-08-26T13:47:32.759 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:4900 IOs, 1750337536 bytes 2023-08-26T13:47:32.961 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:5000 IOs, 1800563712 bytes 2023-08-26T13:47:33.112 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:5100 IOs, 1851738112 bytes 2023-08-26T13:47:33.366 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:5200 IOs, 1912811520 bytes 2023-08-26T13:47:33.492 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:5300 IOs, 1930518528 bytes 2023-08-26T13:47:33.666 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:5400 IOs, 1975665664 bytes 2023-08-26T13:47:33.995 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:5500 IOs, 2082475008 bytes 2023-08-26T13:47:34.244 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:5600 IOs, 2155803648 bytes 2023-08-26T13:47:34.454 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:5700 IOs, 2218712064 bytes 2023-08-26T13:47:34.682 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:5800 IOs, 2287365120 bytes 2023-08-26T13:47:34.793 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:5900 IOs, 2305549312 bytes 2023-08-26T13:47:34.954 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:6000 IOs, 2329520128 bytes 2023-08-26T13:47:35.124 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:6100 IOs, 2364903424 bytes 2023-08-26T13:47:35.324 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:6200 IOs, 2415275008 bytes 2023-08-26T13:47:35.517 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:6300 IOs, 2471411712 bytes 2023-08-26T13:47:35.731 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:6400 IOs, 2516409344 bytes 2023-08-26T13:47:35.905 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:6500 IOs, 2556188672 bytes 2023-08-26T13:47:36.083 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:6600 IOs, 2617380864 bytes 2023-08-26T13:47:36.215 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:6700 IOs, 2618014720 bytes 2023-08-26T13:47:36.473 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:6800 IOs, 2696900608 bytes 2023-08-26T13:47:36.641 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:6900 IOs, 2727912448 bytes 2023-08-26T13:47:36.857 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:7000 IOs, 2792095744 bytes 2023-08-26T13:47:36.993 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:7100 IOs, 2808748032 bytes 2023-08-26T13:47:37.157 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:7200 IOs, 2847269888 bytes 2023-08-26T13:47:37.254 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ error_code=124 2023-08-26T13:47:37.254 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ set -e 2023-08-26T13:47:37.255 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ '[' 124 -eq 124 ']' 2023-08-26T13:47:37.255 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ return 0 2023-08-26T13:47:37.255 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-08-26T13:47:37.255 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ image=image_30 2023-08-26T13:47:37.255 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ create_image_and_enable_mirror cluster2 mirror image_30 snapshot 128M 2023-08-26T13:47:37.256 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster2 2023-08-26T13:47:37.256 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ shift 2023-08-26T13:47:37.256 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T13:47:37.256 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ shift 2023-08-26T13:47:37.256 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=image_30 2023-08-26T13:47:37.257 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ shift 2023-08-26T13:47:37.257 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local mode=snapshot 2023-08-26T13:47:37.257 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ '[' -n snapshot ']' 2023-08-26T13:47:37.257 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ shift 2023-08-26T13:47:37.257 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ create_image cluster2 mirror image_30 128M 2023-08-26T13:47:37.257 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster2 2023-08-26T13:47:37.258 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ shift 2023-08-26T13:47:37.258 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T13:47:37.258 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ shift 2023-08-26T13:47:37.258 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=image_30 2023-08-26T13:47:37.258 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ shift 2023-08-26T13:47:37.259 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local size=128 2023-08-26T13:47:37.259 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ '[' -n 128M ']' 2023-08-26T13:47:37.259 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ size=128M 2023-08-26T13:47:37.259 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ shift 2023-08-26T13:47:37.259 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock mirror/image_30 2023-08-26T13:47:37.301 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ '[' image = image ']' 2023-08-26T13:47:37.301 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ enable_mirror cluster2 mirror image_30 snapshot 2023-08-26T13:47:37.301 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster2 2023-08-26T13:47:37.302 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T13:47:37.302 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=image_30 2023-08-26T13:47:37.302 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local mode=snapshot 2023-08-26T13:47:37.302 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_30 snapshot 2023-08-26T13:47:38.147 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:Mirroring enabled 2023-08-26T13:47:38.155 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rbd --cluster=cluster2 info mirror/image_30 2023-08-26T13:47:38.193 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:rbd image 'image_30': 2023-08-26T13:47:38.193 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: size 128 MiB in 32 objects 2023-08-26T13:47:38.193 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: order 22 (4 MiB objects) 2023-08-26T13:47:38.194 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: snapshot_count: 1 2023-08-26T13:47:38.194 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: id: 13a3deaf015f 2023-08-26T13:47:38.194 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: block_name_prefix: rbd_data.13a3deaf015f 2023-08-26T13:47:38.194 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: format: 2 2023-08-26T13:47:38.194 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: features: layering, exclusive-lock 2023-08-26T13:47:38.195 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: op_features: 2023-08-26T13:47:38.195 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: flags: 2023-08-26T13:47:38.195 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: create_timestamp: Sat Aug 26 13:47:37 2023 2023-08-26T13:47:38.195 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: access_timestamp: Sat Aug 26 13:47:37 2023 2023-08-26T13:47:38.195 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: modify_timestamp: Sat Aug 26 13:47:37 2023 2023-08-26T13:47:38.196 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: mirroring state: enabled 2023-08-26T13:47:38.196 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: mirroring mode: snapshot 2023-08-26T13:47:38.196 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: mirroring global id: 530f9c41-4e08-4e1d-99cd-a782ba07a616 2023-08-26T13:47:38.196 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: mirroring primary: true 2023-08-26T13:47:38.197 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ '[' -n '' ']' 2023-08-26T13:47:38.197 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ stress_write_image cluster2 mirror image_30 2023-08-26T13:47:38.197 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster2 2023-08-26T13:47:38.198 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T13:47:38.198 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=image_30 2023-08-26T13:47:38.198 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2023-08-26T13:47:38.199 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local duration=10 2023-08-26T13:47:38.199 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ set +e 2023-08-26T13:47:38.199 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ timeout 10s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_30 --debug-rbd=20 --debug-journaler=20 2023-08-26T13:47:38.354 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:100 IOs, 13927424 bytes 2023-08-26T13:47:38.575 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:200 IOs, 82974720 bytes 2023-08-26T13:47:38.680 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:300 IOs, 83582976 bytes 2023-08-26T13:47:38.774 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:400 IOs, 84203520 bytes 2023-08-26T13:47:38.877 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:500 IOs, 90395648 bytes 2023-08-26T13:47:39.133 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:600 IOs, 167766016 bytes 2023-08-26T13:47:39.330 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:700 IOs, 218772480 bytes 2023-08-26T13:47:39.541 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:800 IOs, 264548352 bytes 2023-08-26T13:47:39.736 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:900 IOs, 322224128 bytes 2023-08-26T13:47:39.944 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:1000 IOs, 359549952 bytes 2023-08-26T13:47:40.082 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:1100 IOs, 375758848 bytes 2023-08-26T13:47:40.262 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:1200 IOs, 425570304 bytes 2023-08-26T13:47:40.470 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:1300 IOs, 470025216 bytes 2023-08-26T13:47:40.627 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:1400 IOs, 499461120 bytes 2023-08-26T13:47:40.833 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:1500 IOs, 545647616 bytes 2023-08-26T13:47:40.958 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:1600 IOs, 571819008 bytes 2023-08-26T13:47:41.134 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:1700 IOs, 604483584 bytes 2023-08-26T13:47:41.326 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:1800 IOs, 643902464 bytes 2023-08-26T13:47:41.561 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:1900 IOs, 704453632 bytes 2023-08-26T13:47:41.713 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:2000 IOs, 726178816 bytes 2023-08-26T13:47:41.946 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:2100 IOs, 799469568 bytes 2023-08-26T13:47:42.078 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:2200 IOs, 811120640 bytes 2023-08-26T13:47:42.279 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:2300 IOs, 864859136 bytes 2023-08-26T13:47:42.440 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:2400 IOs, 924365824 bytes 2023-08-26T13:47:42.680 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:2500 IOs, 962043904 bytes 2023-08-26T13:47:42.924 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:2600 IOs, 1040953344 bytes 2023-08-26T13:47:43.121 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:2700 IOs, 1097869312 bytes 2023-08-26T13:47:43.288 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:2800 IOs, 1128668160 bytes 2023-08-26T13:47:43.422 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:2900 IOs, 1146631168 bytes 2023-08-26T13:47:43.572 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:3000 IOs, 1178186752 bytes 2023-08-26T13:47:43.776 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:3100 IOs, 1231065088 bytes 2023-08-26T13:47:43.904 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:3200 IOs, 1259323392 bytes 2023-08-26T13:47:44.171 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:3300 IOs, 1340110848 bytes 2023-08-26T13:47:44.474 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:3400 IOs, 1439350784 bytes 2023-08-26T13:47:44.760 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:3500 IOs, 1528783872 bytes 2023-08-26T13:47:45.052 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:3600 IOs, 1608193024 bytes 2023-08-26T13:47:45.268 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:3700 IOs, 1670890496 bytes 2023-08-26T13:47:45.534 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:3800 IOs, 1753324544 bytes 2023-08-26T13:47:45.704 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:3900 IOs, 1788293120 bytes 2023-08-26T13:47:45.898 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:4000 IOs, 1831029760 bytes 2023-08-26T13:47:46.051 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:4100 IOs, 1881891840 bytes 2023-08-26T13:47:46.238 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:4200 IOs, 1910733824 bytes 2023-08-26T13:47:46.514 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:4300 IOs, 2011512832 bytes 2023-08-26T13:47:46.738 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:4400 IOs, 2057753600 bytes 2023-08-26T13:47:46.881 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:4500 IOs, 2082924544 bytes 2023-08-26T13:47:47.080 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:4600 IOs, 2132901888 bytes 2023-08-26T13:47:47.255 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:4700 IOs, 2170446848 bytes 2023-08-26T13:47:47.421 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:4800 IOs, 2201875456 bytes 2023-08-26T13:47:47.602 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:4900 IOs, 2244398080 bytes 2023-08-26T13:47:47.862 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:5000 IOs, 2320123904 bytes 2023-08-26T13:47:48.006 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:5100 IOs, 2343622656 bytes 2023-08-26T13:47:48.138 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:5200 IOs, 2358224896 bytes 2023-08-26T13:47:48.204 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ error_code=124 2023-08-26T13:47:48.205 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ set -e 2023-08-26T13:47:48.205 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ '[' 124 -eq 124 ']' 2023-08-26T13:47:48.205 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ return 0 2023-08-26T13:47:48.205 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-08-26T13:47:48.206 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ image=image_31 2023-08-26T13:47:48.206 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ create_image_and_enable_mirror cluster2 mirror image_31 snapshot 128M 2023-08-26T13:47:48.206 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster2 2023-08-26T13:47:48.207 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ shift 2023-08-26T13:47:48.207 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T13:47:48.207 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ shift 2023-08-26T13:47:48.207 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=image_31 2023-08-26T13:47:48.207 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ shift 2023-08-26T13:47:48.208 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local mode=snapshot 2023-08-26T13:47:48.208 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ '[' -n snapshot ']' 2023-08-26T13:47:48.208 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ shift 2023-08-26T13:47:48.208 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ create_image cluster2 mirror image_31 128M 2023-08-26T13:47:48.208 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster2 2023-08-26T13:47:48.209 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ shift 2023-08-26T13:47:48.209 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T13:47:48.209 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ shift 2023-08-26T13:47:48.209 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=image_31 2023-08-26T13:47:48.209 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ shift 2023-08-26T13:47:48.210 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local size=128 2023-08-26T13:47:48.210 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ '[' -n 128M ']' 2023-08-26T13:47:48.210 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ size=128M 2023-08-26T13:47:48.210 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ shift 2023-08-26T13:47:48.210 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock mirror/image_31 2023-08-26T13:47:48.250 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ '[' image = image ']' 2023-08-26T13:47:48.250 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ enable_mirror cluster2 mirror image_31 snapshot 2023-08-26T13:47:48.251 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster2 2023-08-26T13:47:48.251 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T13:47:48.251 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=image_31 2023-08-26T13:47:48.251 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local mode=snapshot 2023-08-26T13:47:48.251 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_31 snapshot 2023-08-26T13:47:49.141 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:Mirroring enabled 2023-08-26T13:47:49.149 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rbd --cluster=cluster2 info mirror/image_31 2023-08-26T13:47:49.188 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:rbd image 'image_31': 2023-08-26T13:47:49.189 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: size 128 MiB in 32 objects 2023-08-26T13:47:49.189 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: order 22 (4 MiB objects) 2023-08-26T13:47:49.189 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: snapshot_count: 1 2023-08-26T13:47:49.189 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: id: 13ac6d606262 2023-08-26T13:47:49.189 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: block_name_prefix: rbd_data.13ac6d606262 2023-08-26T13:47:49.190 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: format: 2 2023-08-26T13:47:49.190 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: features: layering, exclusive-lock 2023-08-26T13:47:49.190 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: op_features: 2023-08-26T13:47:49.190 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: flags: 2023-08-26T13:47:49.190 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: create_timestamp: Sat Aug 26 13:47:48 2023 2023-08-26T13:47:49.191 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: access_timestamp: Sat Aug 26 13:47:48 2023 2023-08-26T13:47:49.191 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: modify_timestamp: Sat Aug 26 13:47:48 2023 2023-08-26T13:47:49.191 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: mirroring state: enabled 2023-08-26T13:47:49.191 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: mirroring mode: snapshot 2023-08-26T13:47:49.191 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: mirroring global id: c5831e57-7926-4f65-bb0c-ef45cd41b0d4 2023-08-26T13:47:49.192 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: mirroring primary: true 2023-08-26T13:47:49.192 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ '[' -n '' ']' 2023-08-26T13:47:49.193 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ stress_write_image cluster2 mirror image_31 2023-08-26T13:47:49.193 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster2 2023-08-26T13:47:49.193 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T13:47:49.193 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=image_31 2023-08-26T13:47:49.194 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2023-08-26T13:47:49.194 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local duration=8 2023-08-26T13:47:49.194 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ set +e 2023-08-26T13:47:49.195 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ timeout 8s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_31 --debug-rbd=20 --debug-journaler=20 2023-08-26T13:47:49.373 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:100 IOs, 21411840 bytes 2023-08-26T13:47:49.585 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:200 IOs, 82797568 bytes 2023-08-26T13:47:49.808 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:300 IOs, 145227776 bytes 2023-08-26T13:47:49.980 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:400 IOs, 174558208 bytes 2023-08-26T13:47:50.148 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:500 IOs, 205409280 bytes 2023-08-26T13:47:50.243 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:600 IOs, 206016512 bytes 2023-08-26T13:47:50.520 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:700 IOs, 296546304 bytes 2023-08-26T13:47:50.724 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:800 IOs, 337898496 bytes 2023-08-26T13:47:50.866 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:900 IOs, 357502976 bytes 2023-08-26T13:47:51.035 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:1000 IOs, 392198144 bytes 2023-08-26T13:47:51.175 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:1100 IOs, 422718464 bytes 2023-08-26T13:47:51.376 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:1200 IOs, 474323968 bytes 2023-08-26T13:47:51.535 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:1300 IOs, 523934720 bytes 2023-08-26T13:47:51.718 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:1400 IOs, 547968000 bytes 2023-08-26T13:47:51.872 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:1500 IOs, 577154048 bytes 2023-08-26T13:47:52.018 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:1600 IOs, 600968192 bytes 2023-08-26T13:47:52.249 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:1700 IOs, 673512448 bytes 2023-08-26T13:47:52.535 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:1800 IOs, 766158848 bytes 2023-08-26T13:47:52.893 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:1900 IOs, 880577536 bytes 2023-08-26T13:47:53.057 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:2000 IOs, 909997056 bytes 2023-08-26T13:47:53.218 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:2100 IOs, 949264384 bytes 2023-08-26T13:47:53.418 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:2200 IOs, 1007476736 bytes 2023-08-26T13:47:53.622 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:2300 IOs, 1047393280 bytes 2023-08-26T13:47:53.822 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:2400 IOs, 1091399680 bytes 2023-08-26T13:47:54.076 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:2500 IOs, 1145992192 bytes 2023-08-26T13:47:54.316 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:2600 IOs, 1196646400 bytes 2023-08-26T13:47:54.518 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:2700 IOs, 1230195712 bytes 2023-08-26T13:47:54.635 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:2800 IOs, 1237309440 bytes 2023-08-26T13:47:54.840 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:2900 IOs, 1290843136 bytes 2023-08-26T13:47:55.033 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:3000 IOs, 1337295872 bytes 2023-08-26T13:47:55.209 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:3100 IOs, 1369266176 bytes 2023-08-26T13:47:55.343 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:3200 IOs, 1382539264 bytes 2023-08-26T13:47:55.461 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:3300 IOs, 1388908544 bytes 2023-08-26T13:47:55.645 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:3400 IOs, 1422718976 bytes 2023-08-26T13:47:55.754 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:3500 IOs, 1428183040 bytes 2023-08-26T13:47:55.969 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:3600 IOs, 1483947008 bytes 2023-08-26T13:47:56.312 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:3700 IOs, 1600138240 bytes 2023-08-26T13:47:56.500 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.0 with skip (0.3064703687398034) > weight (0.25) 2023-08-26T13:47:56.500 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.1 with skip (0.8595810134502219) > weight (0.25) 2023-08-26T13:47:56.501 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.2 with skip (0.44741970928714236) > weight (0.25) 2023-08-26T13:47:56.501 INFO:tasks.rbd_mirror_thrash:kill cluster2.client.mirror.3 2023-08-26T13:47:56.501 INFO:tasks.rbd_mirror.cluster2.client.mirror.3:Sent signal 15 2023-08-26T13:47:56.502 INFO:tasks.rbd_mirror_thrash:waiting for 4 secs before reviving daemons 2023-08-26T13:47:56.502 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi124.stderr:2023-08-26T13:47:56.501+0000 7fc0ac893700 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.3 (PID: 106146) UID: 1000 2023-08-26T13:47:56.642 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:3800 IOs, 1717182464 bytes 2023-08-26T13:47:56.797 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:3900 IOs, 1740308480 bytes 2023-08-26T13:47:57.045 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:4000 IOs, 1814464512 bytes 2023-08-26T13:47:57.199 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ error_code=124 2023-08-26T13:47:57.199 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ set -e 2023-08-26T13:47:57.199 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ '[' 124 -eq 124 ']' 2023-08-26T13:47:57.200 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ return 0 2023-08-26T13:47:57.200 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-08-26T13:47:57.200 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ image=image_32 2023-08-26T13:47:57.200 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ create_image_and_enable_mirror cluster2 mirror image_32 snapshot 128M 2023-08-26T13:47:57.200 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster2 2023-08-26T13:47:57.201 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ shift 2023-08-26T13:47:57.201 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T13:47:57.201 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ shift 2023-08-26T13:47:57.201 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=image_32 2023-08-26T13:47:57.201 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ shift 2023-08-26T13:47:57.201 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local mode=snapshot 2023-08-26T13:47:57.202 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ '[' -n snapshot ']' 2023-08-26T13:47:57.202 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ shift 2023-08-26T13:47:57.202 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ create_image cluster2 mirror image_32 128M 2023-08-26T13:47:57.202 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster2 2023-08-26T13:47:57.202 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ shift 2023-08-26T13:47:57.202 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T13:47:57.203 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ shift 2023-08-26T13:47:57.203 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=image_32 2023-08-26T13:47:57.203 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ shift 2023-08-26T13:47:57.203 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local size=128 2023-08-26T13:47:57.203 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ '[' -n 128M ']' 2023-08-26T13:47:57.204 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ size=128M 2023-08-26T13:47:57.204 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ shift 2023-08-26T13:47:57.204 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock mirror/image_32 2023-08-26T13:47:57.245 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ '[' image = image ']' 2023-08-26T13:47:57.245 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ enable_mirror cluster2 mirror image_32 snapshot 2023-08-26T13:47:57.245 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster2 2023-08-26T13:47:57.245 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T13:47:57.245 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=image_32 2023-08-26T13:47:57.246 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local mode=snapshot 2023-08-26T13:47:57.246 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_32 snapshot 2023-08-26T13:47:58.154 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:Mirroring enabled 2023-08-26T13:47:58.163 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rbd --cluster=cluster2 info mirror/image_32 2023-08-26T13:47:58.201 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:rbd image 'image_32': 2023-08-26T13:47:58.201 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: size 128 MiB in 32 objects 2023-08-26T13:47:58.201 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: order 22 (4 MiB objects) 2023-08-26T13:47:58.201 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: snapshot_count: 1 2023-08-26T13:47:58.202 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: id: 13b4ce341c8 2023-08-26T13:47:58.202 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: block_name_prefix: rbd_data.13b4ce341c8 2023-08-26T13:47:58.202 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: format: 2 2023-08-26T13:47:58.202 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: features: layering, exclusive-lock 2023-08-26T13:47:58.202 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: op_features: 2023-08-26T13:47:58.203 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: flags: 2023-08-26T13:47:58.203 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: create_timestamp: Sat Aug 26 13:47:57 2023 2023-08-26T13:47:58.203 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: access_timestamp: Sat Aug 26 13:47:57 2023 2023-08-26T13:47:58.203 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: modify_timestamp: Sat Aug 26 13:47:57 2023 2023-08-26T13:47:58.203 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: mirroring state: enabled 2023-08-26T13:47:58.204 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: mirroring mode: snapshot 2023-08-26T13:47:58.204 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: mirroring global id: 9ec01316-ea12-44fa-bbc4-3dc34cb1f619 2023-08-26T13:47:58.204 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: mirroring primary: true 2023-08-26T13:47:58.205 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ '[' -n '' ']' 2023-08-26T13:47:58.205 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ stress_write_image cluster2 mirror image_32 2023-08-26T13:47:58.205 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster2 2023-08-26T13:47:58.205 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T13:47:58.206 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=image_32 2023-08-26T13:47:58.206 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2023-08-26T13:47:58.206 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local duration=5 2023-08-26T13:47:58.207 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ set +e 2023-08-26T13:47:58.207 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ timeout 5s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_32 --debug-rbd=20 --debug-journaler=20 2023-08-26T13:47:58.397 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:100 IOs, 29552640 bytes 2023-08-26T13:47:58.508 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:200 IOs, 34907136 bytes 2023-08-26T13:47:58.744 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:300 IOs, 119840768 bytes 2023-08-26T13:47:58.992 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:400 IOs, 187967488 bytes 2023-08-26T13:47:59.187 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:500 IOs, 254767104 bytes 2023-08-26T13:47:59.446 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:600 IOs, 304563200 bytes 2023-08-26T13:47:59.562 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:700 IOs, 313382912 bytes 2023-08-26T13:47:59.716 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:800 IOs, 341412864 bytes 2023-08-26T13:48:00.022 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:900 IOs, 442982400 bytes 2023-08-26T13:48:00.176 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:1000 IOs, 489235456 bytes 2023-08-26T13:48:00.387 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:1100 IOs, 535867392 bytes 2023-08-26T13:48:00.502 INFO:tasks.rbd_mirror_thrash:waiting for cluster2.client.mirror.3 2023-08-26T13:48:00.503 INFO:teuthology.orchestra.run:waiting for 600 2023-08-26T13:48:00.503 INFO:tasks.rbd_mirror_thrash:reviving cluster2.client.mirror.3 2023-08-26T13:48:00.503 INFO:tasks.rbd_mirror.cluster2.client.mirror.3:Restarting daemon 2023-08-26T13:48:00.504 DEBUG:teuthology.orchestra.run.smithi124:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.3 2023-08-26T13:48:00.508 INFO:tasks.rbd_mirror.cluster2.client.mirror.3:Started 2023-08-26T13:48:00.508 INFO:tasks.rbd_mirror_thrash:waiting for 96 secs before thrashing 2023-08-26T13:48:00.536 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:1200 IOs, 557203456 bytes 2023-08-26T13:48:00.774 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:1300 IOs, 624477184 bytes 2023-08-26T13:48:00.961 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:1400 IOs, 677444608 bytes 2023-08-26T13:48:01.336 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:1500 IOs, 800204800 bytes 2023-08-26T13:48:01.491 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:1600 IOs, 836358144 bytes 2023-08-26T13:48:01.601 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:1700 IOs, 862442496 bytes 2023-08-26T13:48:01.785 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:1800 IOs, 876201984 bytes 2023-08-26T13:48:02.015 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:1900 IOs, 948334592 bytes 2023-08-26T13:48:02.166 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:2000 IOs, 966325248 bytes 2023-08-26T13:48:02.331 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:2100 IOs, 998107136 bytes 2023-08-26T13:48:02.667 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:2200 IOs, 1107878912 bytes 2023-08-26T13:48:02.837 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:2300 IOs, 1136447488 bytes 2023-08-26T13:48:03.030 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:2400 IOs, 1177943040 bytes 2023-08-26T13:48:03.184 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:2500 IOs, 1200241664 bytes 2023-08-26T13:48:03.213 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ error_code=124 2023-08-26T13:48:03.214 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ set -e 2023-08-26T13:48:03.214 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ '[' 124 -eq 124 ']' 2023-08-26T13:48:03.214 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ return 0 2023-08-26T13:48:03.214 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-08-26T13:48:03.215 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ image=image_33 2023-08-26T13:48:03.215 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ create_image_and_enable_mirror cluster2 mirror image_33 snapshot 128M 2023-08-26T13:48:03.215 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster2 2023-08-26T13:48:03.215 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ shift 2023-08-26T13:48:03.215 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T13:48:03.216 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ shift 2023-08-26T13:48:03.216 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=image_33 2023-08-26T13:48:03.216 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ shift 2023-08-26T13:48:03.216 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local mode=snapshot 2023-08-26T13:48:03.216 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ '[' -n snapshot ']' 2023-08-26T13:48:03.216 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ shift 2023-08-26T13:48:03.217 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ create_image cluster2 mirror image_33 128M 2023-08-26T13:48:03.217 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster2 2023-08-26T13:48:03.217 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ shift 2023-08-26T13:48:03.217 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T13:48:03.217 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ shift 2023-08-26T13:48:03.218 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=image_33 2023-08-26T13:48:03.218 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ shift 2023-08-26T13:48:03.218 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local size=128 2023-08-26T13:48:03.218 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ '[' -n 128M ']' 2023-08-26T13:48:03.218 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ size=128M 2023-08-26T13:48:03.219 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ shift 2023-08-26T13:48:03.219 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock mirror/image_33 2023-08-26T13:48:03.259 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ '[' image = image ']' 2023-08-26T13:48:03.259 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ enable_mirror cluster2 mirror image_33 snapshot 2023-08-26T13:48:03.260 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster2 2023-08-26T13:48:03.260 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T13:48:03.260 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=image_33 2023-08-26T13:48:03.260 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local mode=snapshot 2023-08-26T13:48:03.260 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_33 snapshot 2023-08-26T13:48:04.178 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:Mirroring enabled 2023-08-26T13:48:04.185 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rbd --cluster=cluster2 info mirror/image_33 2023-08-26T13:48:04.223 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:rbd image 'image_33': 2023-08-26T13:48:04.223 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: size 128 MiB in 32 objects 2023-08-26T13:48:04.223 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: order 22 (4 MiB objects) 2023-08-26T13:48:04.224 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: snapshot_count: 1 2023-08-26T13:48:04.224 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: id: 13c4564de695 2023-08-26T13:48:04.224 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: block_name_prefix: rbd_data.13c4564de695 2023-08-26T13:48:04.224 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: format: 2 2023-08-26T13:48:04.224 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: features: layering, exclusive-lock 2023-08-26T13:48:04.225 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: op_features: 2023-08-26T13:48:04.225 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: flags: 2023-08-26T13:48:04.225 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: create_timestamp: Sat Aug 26 13:48:03 2023 2023-08-26T13:48:04.225 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: access_timestamp: Sat Aug 26 13:48:03 2023 2023-08-26T13:48:04.225 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: modify_timestamp: Sat Aug 26 13:48:03 2023 2023-08-26T13:48:04.225 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: mirroring state: enabled 2023-08-26T13:48:04.226 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: mirroring mode: snapshot 2023-08-26T13:48:04.226 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: mirroring global id: 3c30bb33-dd5f-4030-adb6-12b16331cd53 2023-08-26T13:48:04.226 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: mirroring primary: true 2023-08-26T13:48:04.227 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ '[' -n '' ']' 2023-08-26T13:48:04.227 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ stress_write_image cluster2 mirror image_33 2023-08-26T13:48:04.227 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster2 2023-08-26T13:48:04.228 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T13:48:04.228 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=image_33 2023-08-26T13:48:04.228 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2023-08-26T13:48:04.229 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local duration=6 2023-08-26T13:48:04.229 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ set +e 2023-08-26T13:48:04.229 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ timeout 6s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_33 --debug-rbd=20 --debug-journaler=20 2023-08-26T13:48:04.444 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:100 IOs, 32779264 bytes 2023-08-26T13:48:04.600 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:200 IOs, 90012672 bytes 2023-08-26T13:48:04.845 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:300 IOs, 128445440 bytes 2023-08-26T13:48:05.105 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:400 IOs, 200799232 bytes 2023-08-26T13:48:05.279 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:500 IOs, 261680128 bytes 2023-08-26T13:48:05.612 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:600 IOs, 350916608 bytes 2023-08-26T13:48:05.843 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:700 IOs, 412502016 bytes 2023-08-26T13:48:06.008 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:800 IOs, 436601856 bytes 2023-08-26T13:48:06.192 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:900 IOs, 478846976 bytes 2023-08-26T13:48:06.323 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:1000 IOs, 502507520 bytes 2023-08-26T13:48:06.511 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:1100 IOs, 534546432 bytes 2023-08-26T13:48:06.671 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:1200 IOs, 564233216 bytes 2023-08-26T13:48:06.809 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:1300 IOs, 581259264 bytes 2023-08-26T13:48:06.913 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:1400 IOs, 594186240 bytes 2023-08-26T13:48:07.039 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:1500 IOs, 597336064 bytes 2023-08-26T13:48:07.205 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:1600 IOs, 645060608 bytes 2023-08-26T13:48:07.353 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:1700 IOs, 667010048 bytes 2023-08-26T13:48:07.503 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:1800 IOs, 686101504 bytes 2023-08-26T13:48:07.705 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:1900 IOs, 739621888 bytes 2023-08-26T13:48:07.808 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:2000 IOs, 740217856 bytes 2023-08-26T13:48:07.958 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:2100 IOs, 765151232 bytes 2023-08-26T13:48:08.117 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:2200 IOs, 815416320 bytes 2023-08-26T13:48:08.293 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:2300 IOs, 832902144 bytes 2023-08-26T13:48:08.461 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:2400 IOs, 866095104 bytes 2023-08-26T13:48:08.609 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:2500 IOs, 884061184 bytes 2023-08-26T13:48:08.794 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:2600 IOs, 921767936 bytes 2023-08-26T13:48:08.939 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:2700 IOs, 945349632 bytes 2023-08-26T13:48:09.138 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:2800 IOs, 1011486720 bytes 2023-08-26T13:48:09.322 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:2900 IOs, 1030554624 bytes 2023-08-26T13:48:09.534 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:3000 IOs, 1089990656 bytes 2023-08-26T13:48:09.778 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:3100 IOs, 1153926144 bytes 2023-08-26T13:48:10.030 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:3200 IOs, 1233386496 bytes 2023-08-26T13:48:10.233 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ error_code=124 2023-08-26T13:48:10.233 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ set -e 2023-08-26T13:48:10.233 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ '[' 124 -eq 124 ']' 2023-08-26T13:48:10.234 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ return 0 2023-08-26T13:48:10.234 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-08-26T13:48:10.234 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ image=image_34 2023-08-26T13:48:10.234 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ create_image_and_enable_mirror cluster2 mirror image_34 snapshot 128M 2023-08-26T13:48:10.235 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster2 2023-08-26T13:48:10.235 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ shift 2023-08-26T13:48:10.235 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T13:48:10.235 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ shift 2023-08-26T13:48:10.235 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=image_34 2023-08-26T13:48:10.236 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ shift 2023-08-26T13:48:10.236 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local mode=snapshot 2023-08-26T13:48:10.236 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ '[' -n snapshot ']' 2023-08-26T13:48:10.236 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ shift 2023-08-26T13:48:10.236 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ create_image cluster2 mirror image_34 128M 2023-08-26T13:48:10.237 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster2 2023-08-26T13:48:10.237 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ shift 2023-08-26T13:48:10.237 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T13:48:10.237 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ shift 2023-08-26T13:48:10.237 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=image_34 2023-08-26T13:48:10.237 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ shift 2023-08-26T13:48:10.238 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local size=128 2023-08-26T13:48:10.238 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ '[' -n 128M ']' 2023-08-26T13:48:10.238 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ size=128M 2023-08-26T13:48:10.238 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ shift 2023-08-26T13:48:10.238 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock mirror/image_34 2023-08-26T13:48:10.278 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ '[' image = image ']' 2023-08-26T13:48:10.278 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ enable_mirror cluster2 mirror image_34 snapshot 2023-08-26T13:48:10.278 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster2 2023-08-26T13:48:10.279 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T13:48:10.279 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=image_34 2023-08-26T13:48:10.279 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local mode=snapshot 2023-08-26T13:48:10.279 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_34 snapshot 2023-08-26T13:48:11.198 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:Mirroring enabled 2023-08-26T13:48:11.206 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rbd --cluster=cluster2 info mirror/image_34 2023-08-26T13:48:11.244 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:rbd image 'image_34': 2023-08-26T13:48:11.244 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: size 128 MiB in 32 objects 2023-08-26T13:48:11.244 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: order 22 (4 MiB objects) 2023-08-26T13:48:11.244 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: snapshot_count: 1 2023-08-26T13:48:11.245 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: id: 13ce4996eeba 2023-08-26T13:48:11.245 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: block_name_prefix: rbd_data.13ce4996eeba 2023-08-26T13:48:11.245 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: format: 2 2023-08-26T13:48:11.245 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: features: layering, exclusive-lock 2023-08-26T13:48:11.245 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: op_features: 2023-08-26T13:48:11.246 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: flags: 2023-08-26T13:48:11.246 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: create_timestamp: Sat Aug 26 13:48:10 2023 2023-08-26T13:48:11.246 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: access_timestamp: Sat Aug 26 13:48:10 2023 2023-08-26T13:48:11.246 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: modify_timestamp: Sat Aug 26 13:48:10 2023 2023-08-26T13:48:11.246 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: mirroring state: enabled 2023-08-26T13:48:11.247 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: mirroring mode: snapshot 2023-08-26T13:48:11.247 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: mirroring global id: ee57f6e1-9b01-446a-9317-2f9969ecc132 2023-08-26T13:48:11.247 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: mirroring primary: true 2023-08-26T13:48:11.248 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ '[' -n '' ']' 2023-08-26T13:48:11.248 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ stress_write_image cluster2 mirror image_34 2023-08-26T13:48:11.248 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster2 2023-08-26T13:48:11.248 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T13:48:11.249 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=image_34 2023-08-26T13:48:11.249 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2023-08-26T13:48:11.249 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local duration=8 2023-08-26T13:48:11.250 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ set +e 2023-08-26T13:48:11.250 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ timeout 8s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_34 --debug-rbd=20 --debug-journaler=20 2023-08-26T13:48:11.461 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:100 IOs, 40235008 bytes 2023-08-26T13:48:11.637 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:200 IOs, 81840128 bytes 2023-08-26T13:48:11.744 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:300 IOs, 101169152 bytes 2023-08-26T13:48:12.012 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:400 IOs, 169803776 bytes 2023-08-26T13:48:12.125 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:500 IOs, 178625536 bytes 2023-08-26T13:48:12.232 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:600 IOs, 182296576 bytes 2023-08-26T13:48:12.389 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:700 IOs, 207214592 bytes 2023-08-26T13:48:12.517 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:800 IOs, 223796224 bytes 2023-08-26T13:48:12.653 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:900 IOs, 244172800 bytes 2023-08-26T13:48:12.810 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:1000 IOs, 268407808 bytes 2023-08-26T13:48:12.970 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:1100 IOs, 300857344 bytes 2023-08-26T13:48:13.149 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:1200 IOs, 343207936 bytes 2023-08-26T13:48:13.327 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:1300 IOs, 381063168 bytes 2023-08-26T13:48:13.444 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:1400 IOs, 381679616 bytes 2023-08-26T13:48:13.583 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:1500 IOs, 395393024 bytes 2023-08-26T13:48:13.758 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:1600 IOs, 426990592 bytes 2023-08-26T13:48:13.914 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:1700 IOs, 456169472 bytes 2023-08-26T13:48:14.116 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:1800 IOs, 509380608 bytes 2023-08-26T13:48:14.332 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:1900 IOs, 558445568 bytes 2023-08-26T13:48:14.585 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:2000 IOs, 624957440 bytes 2023-08-26T13:48:14.784 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:2100 IOs, 670119936 bytes 2023-08-26T13:48:14.977 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:2200 IOs, 712541184 bytes 2023-08-26T13:48:15.123 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:2300 IOs, 733448192 bytes 2023-08-26T13:48:15.309 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:2400 IOs, 782845952 bytes 2023-08-26T13:48:15.505 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:2500 IOs, 832864256 bytes 2023-08-26T13:48:15.703 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:2600 IOs, 873657344 bytes 2023-08-26T13:48:15.889 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:2700 IOs, 928779264 bytes 2023-08-26T13:48:16.187 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:2800 IOs, 1014130688 bytes 2023-08-26T13:48:16.394 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:2900 IOs, 1068375040 bytes 2023-08-26T13:48:16.536 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:3000 IOs, 1113653248 bytes 2023-08-26T13:48:16.764 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:3100 IOs, 1146316800 bytes 2023-08-26T13:48:17.023 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:3200 IOs, 1225722880 bytes 2023-08-26T13:48:17.186 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:3300 IOs, 1256092672 bytes 2023-08-26T13:48:17.344 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:3400 IOs, 1318584320 bytes 2023-08-26T13:48:17.522 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:3500 IOs, 1319195648 bytes 2023-08-26T13:48:17.702 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:3600 IOs, 1373717504 bytes 2023-08-26T13:48:17.877 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:3700 IOs, 1387133952 bytes 2023-08-26T13:48:18.046 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:3800 IOs, 1419189248 bytes 2023-08-26T13:48:18.166 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:3900 IOs, 1425957888 bytes 2023-08-26T13:48:18.445 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:4000 IOs, 1523225600 bytes 2023-08-26T13:48:18.663 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:4100 IOs, 1605569536 bytes 2023-08-26T13:48:18.873 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:4200 IOs, 1638471680 bytes 2023-08-26T13:48:19.077 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:4300 IOs, 1686907904 bytes 2023-08-26T13:48:19.219 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:4400 IOs, 1719268352 bytes 2023-08-26T13:48:19.254 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ error_code=124 2023-08-26T13:48:19.255 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ set -e 2023-08-26T13:48:19.255 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ '[' 124 -eq 124 ']' 2023-08-26T13:48:19.255 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ return 0 2023-08-26T13:48:19.255 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-08-26T13:48:19.256 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ image=image_35 2023-08-26T13:48:19.256 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ create_image_and_enable_mirror cluster2 mirror image_35 snapshot 128M 2023-08-26T13:48:19.256 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster2 2023-08-26T13:48:19.256 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ shift 2023-08-26T13:48:19.256 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T13:48:19.256 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ shift 2023-08-26T13:48:19.257 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=image_35 2023-08-26T13:48:19.257 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ shift 2023-08-26T13:48:19.257 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local mode=snapshot 2023-08-26T13:48:19.257 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ '[' -n snapshot ']' 2023-08-26T13:48:19.257 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ shift 2023-08-26T13:48:19.258 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ create_image cluster2 mirror image_35 128M 2023-08-26T13:48:19.258 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster2 2023-08-26T13:48:19.258 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ shift 2023-08-26T13:48:19.258 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T13:48:19.258 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ shift 2023-08-26T13:48:19.259 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=image_35 2023-08-26T13:48:19.259 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ shift 2023-08-26T13:48:19.259 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local size=128 2023-08-26T13:48:19.259 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ '[' -n 128M ']' 2023-08-26T13:48:19.259 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ size=128M 2023-08-26T13:48:19.259 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ shift 2023-08-26T13:48:19.260 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock mirror/image_35 2023-08-26T13:48:19.301 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ '[' image = image ']' 2023-08-26T13:48:19.302 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ enable_mirror cluster2 mirror image_35 snapshot 2023-08-26T13:48:19.302 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster2 2023-08-26T13:48:19.302 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T13:48:19.302 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=image_35 2023-08-26T13:48:19.302 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local mode=snapshot 2023-08-26T13:48:19.303 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_35 snapshot 2023-08-26T13:48:20.152 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:Mirroring enabled 2023-08-26T13:48:20.160 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rbd --cluster=cluster2 info mirror/image_35 2023-08-26T13:48:20.196 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:rbd image 'image_35': 2023-08-26T13:48:20.197 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: size 128 MiB in 32 objects 2023-08-26T13:48:20.197 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: order 22 (4 MiB objects) 2023-08-26T13:48:20.197 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: snapshot_count: 1 2023-08-26T13:48:20.197 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: id: 13d720bea9ea 2023-08-26T13:48:20.197 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: block_name_prefix: rbd_data.13d720bea9ea 2023-08-26T13:48:20.198 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: format: 2 2023-08-26T13:48:20.198 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: features: layering, exclusive-lock 2023-08-26T13:48:20.198 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: op_features: 2023-08-26T13:48:20.198 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: flags: 2023-08-26T13:48:20.198 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: create_timestamp: Sat Aug 26 13:48:19 2023 2023-08-26T13:48:20.199 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: access_timestamp: Sat Aug 26 13:48:19 2023 2023-08-26T13:48:20.199 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: modify_timestamp: Sat Aug 26 13:48:19 2023 2023-08-26T13:48:20.199 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: mirroring state: enabled 2023-08-26T13:48:20.199 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: mirroring mode: snapshot 2023-08-26T13:48:20.199 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: mirroring global id: 92367d6b-5947-4538-a25d-c7bdd622153a 2023-08-26T13:48:20.200 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: mirroring primary: true 2023-08-26T13:48:20.200 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ '[' -n '' ']' 2023-08-26T13:48:20.201 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ stress_write_image cluster2 mirror image_35 2023-08-26T13:48:20.201 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster2 2023-08-26T13:48:20.201 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T13:48:20.201 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=image_35 2023-08-26T13:48:20.201 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2023-08-26T13:48:20.202 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local duration=13 2023-08-26T13:48:20.202 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ set +e 2023-08-26T13:48:20.203 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ timeout 13s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_35 --debug-rbd=20 --debug-journaler=20 2023-08-26T13:48:20.407 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:100 IOs, 31502336 bytes 2023-08-26T13:48:20.507 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:200 IOs, 32141312 bytes 2023-08-26T13:48:20.669 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:300 IOs, 61466624 bytes 2023-08-26T13:48:20.928 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:400 IOs, 143249408 bytes 2023-08-26T13:48:21.148 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:500 IOs, 187683840 bytes 2023-08-26T13:48:21.360 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:600 IOs, 243654656 bytes 2023-08-26T13:48:21.493 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:700 IOs, 258978816 bytes 2023-08-26T13:48:21.715 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:800 IOs, 318788608 bytes 2023-08-26T13:48:21.924 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:900 IOs, 387532800 bytes 2023-08-26T13:48:22.156 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:1000 IOs, 428620800 bytes 2023-08-26T13:48:22.333 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:1100 IOs, 462996480 bytes 2023-08-26T13:48:22.474 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:1200 IOs, 484030464 bytes 2023-08-26T13:48:22.628 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:1300 IOs, 512105472 bytes 2023-08-26T13:48:22.922 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:1400 IOs, 631104512 bytes 2023-08-26T13:48:23.155 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:1500 IOs, 668212224 bytes 2023-08-26T13:48:23.278 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:1600 IOs, 677246976 bytes 2023-08-26T13:48:23.444 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:1700 IOs, 702255104 bytes 2023-08-26T13:48:23.637 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:1800 IOs, 738118656 bytes 2023-08-26T13:48:23.980 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:1900 IOs, 757724160 bytes 2023-08-26T13:48:24.197 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:2000 IOs, 799498240 bytes 2023-08-26T13:48:24.308 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:2100 IOs, 808527872 bytes 2023-08-26T13:48:24.475 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:2200 IOs, 838918144 bytes 2023-08-26T13:48:24.607 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:2300 IOs, 849821696 bytes 2023-08-26T13:48:24.792 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:2400 IOs, 884444160 bytes 2023-08-26T13:48:24.959 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:2500 IOs, 932447232 bytes 2023-08-26T13:48:25.099 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:2600 IOs, 933044224 bytes 2023-08-26T13:48:25.233 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:2700 IOs, 954557440 bytes 2023-08-26T13:48:25.537 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:2800 IOs, 1051670528 bytes 2023-08-26T13:48:25.770 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:2900 IOs, 1128689664 bytes 2023-08-26T13:48:25.943 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:3000 IOs, 1158548480 bytes 2023-08-26T13:48:26.171 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:3100 IOs, 1217449984 bytes 2023-08-26T13:48:26.421 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:3200 IOs, 1310963712 bytes 2023-08-26T13:48:26.668 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:3300 IOs, 1373018112 bytes 2023-08-26T13:48:26.944 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:3400 IOs, 1457896448 bytes 2023-08-26T13:48:27.128 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:3500 IOs, 1499013120 bytes 2023-08-26T13:48:27.252 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:3600 IOs, 1523479552 bytes 2023-08-26T13:48:27.421 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:3700 IOs, 1537824768 bytes 2023-08-26T13:48:27.617 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:3800 IOs, 1588843520 bytes 2023-08-26T13:48:27.837 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:3900 IOs, 1641044992 bytes 2023-08-26T13:48:28.074 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:4000 IOs, 1704681472 bytes 2023-08-26T13:48:28.272 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:4100 IOs, 1764709376 bytes 2023-08-26T13:48:28.444 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:4200 IOs, 1784351744 bytes 2023-08-26T13:48:28.598 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:4300 IOs, 1825925120 bytes 2023-08-26T13:48:28.792 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:4400 IOs, 1853737984 bytes 2023-08-26T13:48:28.923 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:4500 IOs, 1862772736 bytes 2023-08-26T13:48:29.123 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:4600 IOs, 1936063488 bytes 2023-08-26T13:48:29.328 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:4700 IOs, 1962964992 bytes 2023-08-26T13:48:29.499 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:4800 IOs, 1996283904 bytes 2023-08-26T13:48:29.672 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:4900 IOs, 2032524288 bytes 2023-08-26T13:48:29.933 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:5000 IOs, 2110803968 bytes 2023-08-26T13:48:30.051 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:5100 IOs, 2120357888 bytes 2023-08-26T13:48:30.280 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:5200 IOs, 2175450112 bytes 2023-08-26T13:48:30.424 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:5300 IOs, 2210767872 bytes 2023-08-26T13:48:30.661 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:5400 IOs, 2255502336 bytes 2023-08-26T13:48:30.849 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:5500 IOs, 2293788672 bytes 2023-08-26T13:48:30.960 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:5600 IOs, 2295475200 bytes 2023-08-26T13:48:31.175 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:5700 IOs, 2362174464 bytes 2023-08-26T13:48:31.312 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:5800 IOs, 2369156096 bytes 2023-08-26T13:48:31.478 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:5900 IOs, 2399016960 bytes 2023-08-26T13:48:31.711 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:6000 IOs, 2472964096 bytes 2023-08-26T13:48:31.827 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:6100 IOs, 2474259456 bytes 2023-08-26T13:48:32.078 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:6200 IOs, 2546115584 bytes 2023-08-26T13:48:32.243 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:6300 IOs, 2582205440 bytes 2023-08-26T13:48:32.391 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:6400 IOs, 2598870016 bytes 2023-08-26T13:48:32.520 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:6500 IOs, 2611615744 bytes 2023-08-26T13:48:32.625 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:6600 IOs, 2614650880 bytes 2023-08-26T13:48:32.811 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:6700 IOs, 2672724992 bytes 2023-08-26T13:48:33.003 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:6800 IOs, 2702129152 bytes 2023-08-26T13:48:33.209 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ error_code=124 2023-08-26T13:48:33.209 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ set -e 2023-08-26T13:48:33.210 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ '[' 124 -eq 124 ']' 2023-08-26T13:48:33.210 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ return 0 2023-08-26T13:48:33.210 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-08-26T13:48:33.210 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ image=image_36 2023-08-26T13:48:33.210 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ create_image_and_enable_mirror cluster2 mirror image_36 snapshot 128M 2023-08-26T13:48:33.211 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster2 2023-08-26T13:48:33.211 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ shift 2023-08-26T13:48:33.211 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T13:48:33.211 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ shift 2023-08-26T13:48:33.211 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=image_36 2023-08-26T13:48:33.212 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ shift 2023-08-26T13:48:33.212 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local mode=snapshot 2023-08-26T13:48:33.212 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ '[' -n snapshot ']' 2023-08-26T13:48:33.212 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ shift 2023-08-26T13:48:33.212 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ create_image cluster2 mirror image_36 128M 2023-08-26T13:48:33.213 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster2 2023-08-26T13:48:33.213 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ shift 2023-08-26T13:48:33.213 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T13:48:33.213 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ shift 2023-08-26T13:48:33.213 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=image_36 2023-08-26T13:48:33.214 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ shift 2023-08-26T13:48:33.214 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local size=128 2023-08-26T13:48:33.214 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ '[' -n 128M ']' 2023-08-26T13:48:33.214 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ size=128M 2023-08-26T13:48:33.214 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ shift 2023-08-26T13:48:33.215 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock mirror/image_36 2023-08-26T13:48:33.255 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ '[' image = image ']' 2023-08-26T13:48:33.255 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ enable_mirror cluster2 mirror image_36 snapshot 2023-08-26T13:48:33.256 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster2 2023-08-26T13:48:33.256 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T13:48:33.256 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=image_36 2023-08-26T13:48:33.256 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local mode=snapshot 2023-08-26T13:48:33.256 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_36 snapshot 2023-08-26T13:48:34.175 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:Mirroring enabled 2023-08-26T13:48:34.183 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rbd --cluster=cluster2 info mirror/image_36 2023-08-26T13:48:34.223 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:rbd image 'image_36': 2023-08-26T13:48:34.223 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: size 128 MiB in 32 objects 2023-08-26T13:48:34.223 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: order 22 (4 MiB objects) 2023-08-26T13:48:34.224 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: snapshot_count: 1 2023-08-26T13:48:34.224 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: id: 13e01abb5048 2023-08-26T13:48:34.224 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: block_name_prefix: rbd_data.13e01abb5048 2023-08-26T13:48:34.224 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: format: 2 2023-08-26T13:48:34.224 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: features: layering, exclusive-lock 2023-08-26T13:48:34.224 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: op_features: 2023-08-26T13:48:34.225 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: flags: 2023-08-26T13:48:34.225 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: create_timestamp: Sat Aug 26 13:48:33 2023 2023-08-26T13:48:34.225 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: access_timestamp: Sat Aug 26 13:48:33 2023 2023-08-26T13:48:34.225 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: modify_timestamp: Sat Aug 26 13:48:33 2023 2023-08-26T13:48:34.225 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: mirroring state: enabled 2023-08-26T13:48:34.226 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: mirroring mode: snapshot 2023-08-26T13:48:34.226 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: mirroring global id: 8068d21b-b929-4466-9d32-bbd96d0ca61c 2023-08-26T13:48:34.226 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: mirroring primary: true 2023-08-26T13:48:34.227 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ '[' -n '' ']' 2023-08-26T13:48:34.227 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ stress_write_image cluster2 mirror image_36 2023-08-26T13:48:34.227 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster2 2023-08-26T13:48:34.228 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T13:48:34.228 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=image_36 2023-08-26T13:48:34.228 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2023-08-26T13:48:34.229 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local duration=14 2023-08-26T13:48:34.229 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ set +e 2023-08-26T13:48:34.229 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ timeout 14s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_36 --debug-rbd=20 --debug-journaler=20 2023-08-26T13:48:34.504 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:100 IOs, 88146944 bytes 2023-08-26T13:48:34.675 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:200 IOs, 102782976 bytes 2023-08-26T13:48:34.848 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:300 IOs, 133444608 bytes 2023-08-26T13:48:34.988 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:400 IOs, 154526720 bytes 2023-08-26T13:48:35.111 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:500 IOs, 167487488 bytes 2023-08-26T13:48:35.339 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:600 IOs, 231593984 bytes 2023-08-26T13:48:35.487 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:700 IOs, 255934464 bytes 2023-08-26T13:48:35.622 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:800 IOs, 274461696 bytes 2023-08-26T13:48:35.806 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:900 IOs, 311277568 bytes 2023-08-26T13:48:35.949 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:1000 IOs, 332719104 bytes 2023-08-26T13:48:36.143 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:1100 IOs, 380282880 bytes 2023-08-26T13:48:36.381 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:1200 IOs, 443056128 bytes 2023-08-26T13:48:36.563 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:1300 IOs, 485098496 bytes 2023-08-26T13:48:36.703 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:1400 IOs, 508807168 bytes 2023-08-26T13:48:36.840 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:1500 IOs, 519646208 bytes 2023-08-26T13:48:37.102 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:1600 IOs, 595723264 bytes 2023-08-26T13:48:37.252 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:1700 IOs, 615502848 bytes 2023-08-26T13:48:37.531 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:1800 IOs, 705891328 bytes 2023-08-26T13:48:37.733 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:1900 IOs, 757025792 bytes 2023-08-26T13:48:37.929 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:2000 IOs, 802675712 bytes 2023-08-26T13:48:38.076 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:2100 IOs, 826409984 bytes 2023-08-26T13:48:38.275 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:2200 IOs, 877038592 bytes 2023-08-26T13:48:38.503 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:2300 IOs, 942209024 bytes 2023-08-26T13:48:38.824 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:2400 IOs, 1037277184 bytes 2023-08-26T13:48:38.984 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:2500 IOs, 1066822656 bytes 2023-08-26T13:48:39.131 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:2600 IOs, 1090386944 bytes 2023-08-26T13:48:39.309 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:2700 IOs, 1133733888 bytes 2023-08-26T13:48:39.460 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:2800 IOs, 1162427392 bytes 2023-08-26T13:48:39.640 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:2900 IOs, 1204500480 bytes 2023-08-26T13:48:39.802 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:3000 IOs, 1237714944 bytes 2023-08-26T13:48:39.984 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:3100 IOs, 1280201728 bytes 2023-08-26T13:48:40.151 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:3200 IOs, 1307480064 bytes 2023-08-26T13:48:40.398 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:3300 IOs, 1381321728 bytes 2023-08-26T13:48:40.672 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:3400 IOs, 1464603648 bytes 2023-08-26T13:48:40.947 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:3500 IOs, 1550956544 bytes 2023-08-26T13:48:41.092 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:3600 IOs, 1566125056 bytes 2023-08-26T13:48:41.307 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:3700 IOs, 1629557760 bytes 2023-08-26T13:48:41.733 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:3800 IOs, 1800857600 bytes 2023-08-26T13:48:41.948 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:3900 IOs, 1856743424 bytes 2023-08-26T13:48:42.070 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:4000 IOs, 1869877248 bytes 2023-08-26T13:48:42.313 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:4100 IOs, 1937960960 bytes 2023-08-26T13:48:42.470 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:4200 IOs, 1972527104 bytes 2023-08-26T13:48:42.580 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:4300 IOs, 1980457984 bytes 2023-08-26T13:48:42.747 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:4400 IOs, 2027126784 bytes 2023-08-26T13:48:42.990 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:4500 IOs, 2089786368 bytes 2023-08-26T13:48:43.148 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:4600 IOs, 2116179968 bytes 2023-08-26T13:48:43.308 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:4700 IOs, 2143907840 bytes 2023-08-26T13:48:43.434 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:4800 IOs, 2156083200 bytes 2023-08-26T13:48:43.588 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:4900 IOs, 2182408192 bytes 2023-08-26T13:48:43.763 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:5000 IOs, 2213993472 bytes 2023-08-26T13:48:43.925 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:5100 IOs, 2243096576 bytes 2023-08-26T13:48:44.064 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:5200 IOs, 2260813824 bytes 2023-08-26T13:48:44.234 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:5300 IOs, 2295844864 bytes 2023-08-26T13:48:44.452 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:5400 IOs, 2354164736 bytes 2023-08-26T13:48:44.607 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:5500 IOs, 2380059648 bytes 2023-08-26T13:48:44.857 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:5600 IOs, 2445363200 bytes 2023-08-26T13:48:45.112 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:5700 IOs, 2511197184 bytes 2023-08-26T13:48:45.372 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:5800 IOs, 2593888256 bytes 2023-08-26T13:48:45.545 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:5900 IOs, 2623601664 bytes 2023-08-26T13:48:45.728 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:6000 IOs, 2663789568 bytes 2023-08-26T13:48:45.935 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:6100 IOs, 2708554752 bytes 2023-08-26T13:48:46.046 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:6200 IOs, 2709167104 bytes 2023-08-26T13:48:46.220 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:6300 IOs, 2760381440 bytes 2023-08-26T13:48:46.408 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:6400 IOs, 2806854656 bytes 2023-08-26T13:48:46.576 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:6500 IOs, 2822645760 bytes 2023-08-26T13:48:46.834 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:6600 IOs, 2897952768 bytes 2023-08-26T13:48:47.063 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:6700 IOs, 2956351488 bytes 2023-08-26T13:48:47.246 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:6800 IOs, 2991992832 bytes 2023-08-26T13:48:47.258 INFO:tasks.ceph.cluster2.osd.0.smithi124.stderr:2023-08-26T13:48:47.257+0000 7fb60dfa7700 -1 reset not still connected to 0x56452b51f450 2023-08-26T13:48:47.417 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:6900 IOs, 3026787328 bytes 2023-08-26T13:48:47.649 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:7000 IOs, 3086084096 bytes 2023-08-26T13:48:47.881 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:7100 IOs, 3151907840 bytes 2023-08-26T13:48:48.034 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:7200 IOs, 3174650880 bytes 2023-08-26T13:48:48.227 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:7300 IOs, 3216261120 bytes 2023-08-26T13:48:48.234 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ error_code=124 2023-08-26T13:48:48.235 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ set -e 2023-08-26T13:48:48.235 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ '[' 124 -eq 124 ']' 2023-08-26T13:48:48.235 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ return 0 2023-08-26T13:48:48.235 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-08-26T13:48:48.235 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ image=image_37 2023-08-26T13:48:48.236 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ create_image_and_enable_mirror cluster2 mirror image_37 snapshot 128M 2023-08-26T13:48:48.236 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster2 2023-08-26T13:48:48.236 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ shift 2023-08-26T13:48:48.236 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T13:48:48.236 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ shift 2023-08-26T13:48:48.237 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=image_37 2023-08-26T13:48:48.237 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ shift 2023-08-26T13:48:48.237 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local mode=snapshot 2023-08-26T13:48:48.237 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ '[' -n snapshot ']' 2023-08-26T13:48:48.237 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ shift 2023-08-26T13:48:48.238 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ create_image cluster2 mirror image_37 128M 2023-08-26T13:48:48.238 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster2 2023-08-26T13:48:48.238 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ shift 2023-08-26T13:48:48.238 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T13:48:48.238 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ shift 2023-08-26T13:48:48.239 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=image_37 2023-08-26T13:48:48.239 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ shift 2023-08-26T13:48:48.239 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local size=128 2023-08-26T13:48:48.239 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ '[' -n 128M ']' 2023-08-26T13:48:48.239 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ size=128M 2023-08-26T13:48:48.239 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ shift 2023-08-26T13:48:48.240 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock mirror/image_37 2023-08-26T13:48:48.280 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ '[' image = image ']' 2023-08-26T13:48:48.280 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ enable_mirror cluster2 mirror image_37 snapshot 2023-08-26T13:48:48.280 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster2 2023-08-26T13:48:48.281 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T13:48:48.281 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=image_37 2023-08-26T13:48:48.281 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local mode=snapshot 2023-08-26T13:48:48.281 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_37 snapshot 2023-08-26T13:48:49.371 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:Mirroring enabled 2023-08-26T13:48:49.380 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rbd --cluster=cluster2 info mirror/image_37 2023-08-26T13:48:49.419 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:rbd image 'image_37': 2023-08-26T13:48:49.420 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: size 128 MiB in 32 objects 2023-08-26T13:48:49.420 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: order 22 (4 MiB objects) 2023-08-26T13:48:49.420 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: snapshot_count: 1 2023-08-26T13:48:49.421 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: id: 13e882836089 2023-08-26T13:48:49.421 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: block_name_prefix: rbd_data.13e882836089 2023-08-26T13:48:49.421 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: format: 2 2023-08-26T13:48:49.421 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: features: layering, exclusive-lock 2023-08-26T13:48:49.421 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: op_features: 2023-08-26T13:48:49.422 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: flags: 2023-08-26T13:48:49.422 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: create_timestamp: Sat Aug 26 13:48:48 2023 2023-08-26T13:48:49.422 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: access_timestamp: Sat Aug 26 13:48:48 2023 2023-08-26T13:48:49.422 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: modify_timestamp: Sat Aug 26 13:48:48 2023 2023-08-26T13:48:49.422 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: mirroring state: enabled 2023-08-26T13:48:49.423 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: mirroring mode: snapshot 2023-08-26T13:48:49.423 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: mirroring global id: 339b230b-a92d-47ff-8812-3e7a71f8b968 2023-08-26T13:48:49.423 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: mirroring primary: true 2023-08-26T13:48:49.424 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ '[' -n '' ']' 2023-08-26T13:48:49.424 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ stress_write_image cluster2 mirror image_37 2023-08-26T13:48:49.425 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster2 2023-08-26T13:48:49.425 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T13:48:49.425 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=image_37 2023-08-26T13:48:49.426 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2023-08-26T13:48:49.426 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local duration=6 2023-08-26T13:48:49.426 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ set +e 2023-08-26T13:48:49.426 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ timeout 6s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_37 --debug-rbd=20 --debug-journaler=20 2023-08-26T13:48:49.580 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:100 IOs, 9682944 bytes 2023-08-26T13:48:49.705 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:200 IOs, 26149888 bytes 2023-08-26T13:48:49.858 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:300 IOs, 48226304 bytes 2023-08-26T13:48:50.063 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:400 IOs, 111434752 bytes 2023-08-26T13:48:50.302 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:500 IOs, 177737728 bytes 2023-08-26T13:48:50.551 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:600 IOs, 245412864 bytes 2023-08-26T13:48:50.742 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:700 IOs, 290758656 bytes 2023-08-26T13:48:50.976 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:800 IOs, 352162816 bytes 2023-08-26T13:48:51.204 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:900 IOs, 407914496 bytes 2023-08-26T13:48:51.306 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:1000 IOs, 408537088 bytes 2023-08-26T13:48:51.491 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:1100 IOs, 437729280 bytes 2023-08-26T13:48:51.655 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:1200 IOs, 464892928 bytes 2023-08-26T13:48:51.861 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:1300 IOs, 518994944 bytes 2023-08-26T13:48:52.073 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:1400 IOs, 574386176 bytes 2023-08-26T13:48:52.386 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:1500 IOs, 672110592 bytes 2023-08-26T13:48:52.564 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:1600 IOs, 677139456 bytes 2023-08-26T13:48:52.755 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:1700 IOs, 727174144 bytes 2023-08-26T13:48:52.942 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:1800 IOs, 737572864 bytes 2023-08-26T13:48:53.117 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:1900 IOs, 773429248 bytes 2023-08-26T13:48:53.272 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:2000 IOs, 802134016 bytes 2023-08-26T13:48:53.534 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:2100 IOs, 878828544 bytes 2023-08-26T13:48:53.758 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:2200 IOs, 906378240 bytes 2023-08-26T13:48:54.002 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:2300 IOs, 980187136 bytes 2023-08-26T13:48:54.189 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:2400 IOs, 1006901248 bytes 2023-08-26T13:48:54.367 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:2500 IOs, 1043674112 bytes 2023-08-26T13:48:54.562 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:2600 IOs, 1094290432 bytes 2023-08-26T13:48:54.778 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:2700 IOs, 1151121408 bytes 2023-08-26T13:48:55.011 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:2800 IOs, 1205828608 bytes 2023-08-26T13:48:55.253 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:2900 IOs, 1272072192 bytes 2023-08-26T13:48:55.430 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ error_code=124 2023-08-26T13:48:55.430 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ set -e 2023-08-26T13:48:55.431 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ '[' 124 -eq 124 ']' 2023-08-26T13:48:55.431 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ return 0 2023-08-26T13:48:55.431 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-08-26T13:48:55.431 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ image=image_38 2023-08-26T13:48:55.431 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ create_image_and_enable_mirror cluster2 mirror image_38 snapshot 128M 2023-08-26T13:48:55.432 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster2 2023-08-26T13:48:55.432 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ shift 2023-08-26T13:48:55.432 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T13:48:55.432 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ shift 2023-08-26T13:48:55.432 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=image_38 2023-08-26T13:48:55.432 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ shift 2023-08-26T13:48:55.433 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local mode=snapshot 2023-08-26T13:48:55.433 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ '[' -n snapshot ']' 2023-08-26T13:48:55.433 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ shift 2023-08-26T13:48:55.433 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ create_image cluster2 mirror image_38 128M 2023-08-26T13:48:55.433 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster2 2023-08-26T13:48:55.434 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ shift 2023-08-26T13:48:55.434 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T13:48:55.434 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ shift 2023-08-26T13:48:55.434 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=image_38 2023-08-26T13:48:55.434 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ shift 2023-08-26T13:48:55.434 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local size=128 2023-08-26T13:48:55.435 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ '[' -n 128M ']' 2023-08-26T13:48:55.435 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ size=128M 2023-08-26T13:48:55.435 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ shift 2023-08-26T13:48:55.435 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock mirror/image_38 2023-08-26T13:48:55.476 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ '[' image = image ']' 2023-08-26T13:48:55.476 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ enable_mirror cluster2 mirror image_38 snapshot 2023-08-26T13:48:55.476 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster2 2023-08-26T13:48:55.477 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T13:48:55.477 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=image_38 2023-08-26T13:48:55.477 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local mode=snapshot 2023-08-26T13:48:55.477 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_38 snapshot 2023-08-26T13:48:56.045 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:Mirroring enabled 2023-08-26T13:48:56.053 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rbd --cluster=cluster2 info mirror/image_38 2023-08-26T13:48:56.090 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:rbd image 'image_38': 2023-08-26T13:48:56.090 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: size 128 MiB in 32 objects 2023-08-26T13:48:56.091 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: order 22 (4 MiB objects) 2023-08-26T13:48:56.091 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: snapshot_count: 1 2023-08-26T13:48:56.091 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: id: 13f0d277ae54 2023-08-26T13:48:56.092 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: block_name_prefix: rbd_data.13f0d277ae54 2023-08-26T13:48:56.092 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: format: 2 2023-08-26T13:48:56.092 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: features: layering, exclusive-lock 2023-08-26T13:48:56.092 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: op_features: 2023-08-26T13:48:56.092 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: flags: 2023-08-26T13:48:56.093 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: create_timestamp: Sat Aug 26 13:48:55 2023 2023-08-26T13:48:56.093 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: access_timestamp: Sat Aug 26 13:48:55 2023 2023-08-26T13:48:56.093 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: modify_timestamp: Sat Aug 26 13:48:55 2023 2023-08-26T13:48:56.093 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: mirroring state: enabled 2023-08-26T13:48:56.094 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: mirroring mode: snapshot 2023-08-26T13:48:56.094 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: mirroring global id: 1e20d016-f50e-43c9-b2e8-47e995fee9bd 2023-08-26T13:48:56.094 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: mirroring primary: true 2023-08-26T13:48:56.095 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ '[' -n '' ']' 2023-08-26T13:48:56.095 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ stress_write_image cluster2 mirror image_38 2023-08-26T13:48:56.095 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster2 2023-08-26T13:48:56.096 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T13:48:56.096 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=image_38 2023-08-26T13:48:56.096 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2023-08-26T13:48:56.096 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local duration=9 2023-08-26T13:48:56.096 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ set +e 2023-08-26T13:48:56.097 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ timeout 9s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_38 --debug-rbd=20 --debug-journaler=20 2023-08-26T13:48:56.265 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:100 IOs, 14010368 bytes 2023-08-26T13:48:56.438 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:200 IOs, 58027008 bytes 2023-08-26T13:48:56.648 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:300 IOs, 110297088 bytes 2023-08-26T13:48:56.776 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:400 IOs, 122694656 bytes 2023-08-26T13:48:56.967 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:500 IOs, 171744256 bytes 2023-08-26T13:48:57.142 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:600 IOs, 203445248 bytes 2023-08-26T13:48:57.415 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:700 IOs, 290513920 bytes 2023-08-26T13:48:57.553 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:800 IOs, 312849408 bytes 2023-08-26T13:48:57.693 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:900 IOs, 330414080 bytes 2023-08-26T13:48:57.860 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:1000 IOs, 348638208 bytes 2023-08-26T13:48:58.035 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:1100 IOs, 375439360 bytes 2023-08-26T13:48:58.275 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:1200 IOs, 439513088 bytes 2023-08-26T13:48:58.443 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:1300 IOs, 470197248 bytes 2023-08-26T13:48:58.631 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:1400 IOs, 515054592 bytes 2023-08-26T13:48:58.780 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:1500 IOs, 538368000 bytes 2023-08-26T13:48:58.955 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:1600 IOs, 575521792 bytes 2023-08-26T13:48:59.145 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:1700 IOs, 614998016 bytes 2023-08-26T13:48:59.289 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:1800 IOs, 628116480 bytes 2023-08-26T13:48:59.410 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:1900 IOs, 644454400 bytes 2023-08-26T13:48:59.586 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:2000 IOs, 675885056 bytes 2023-08-26T13:48:59.767 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:2100 IOs, 715260928 bytes 2023-08-26T13:48:59.905 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:2200 IOs, 732211200 bytes 2023-08-26T13:49:00.103 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:2300 IOs, 778571776 bytes 2023-08-26T13:49:00.305 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:2400 IOs, 825341952 bytes 2023-08-26T13:49:00.508 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:2500 IOs, 872315904 bytes 2023-08-26T13:49:00.750 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:2600 IOs, 940545024 bytes 2023-08-26T13:49:00.943 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:2700 IOs, 983394304 bytes 2023-08-26T13:49:01.154 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:2800 IOs, 1034360832 bytes 2023-08-26T13:49:01.298 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:2900 IOs, 1075443712 bytes 2023-08-26T13:49:01.587 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:3000 IOs, 1145744384 bytes 2023-08-26T13:49:01.797 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:3100 IOs, 1200599040 bytes 2023-08-26T13:49:01.981 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:3200 IOs, 1242095616 bytes 2023-08-26T13:49:02.356 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:3300 IOs, 1383371776 bytes 2023-08-26T13:49:02.584 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:3400 IOs, 1469773824 bytes 2023-08-26T13:49:02.779 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:3500 IOs, 1528592384 bytes 2023-08-26T13:49:02.953 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:3600 IOs, 1546136576 bytes 2023-08-26T13:49:03.158 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:3700 IOs, 1579092992 bytes 2023-08-26T13:49:03.296 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:3800 IOs, 1615089664 bytes 2023-08-26T13:49:03.618 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:3900 IOs, 1706960896 bytes 2023-08-26T13:49:03.759 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:4000 IOs, 1720535040 bytes 2023-08-26T13:49:03.877 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:4100 IOs, 1752264704 bytes 2023-08-26T13:49:04.056 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:4200 IOs, 1766085632 bytes 2023-08-26T13:49:04.184 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:4300 IOs, 1799361536 bytes 2023-08-26T13:49:04.418 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:4400 IOs, 1834749952 bytes 2023-08-26T13:49:04.646 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:4500 IOs, 1885049856 bytes 2023-08-26T13:49:04.815 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:4600 IOs, 1918137344 bytes 2023-08-26T13:49:04.926 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:4700 IOs, 1923738624 bytes 2023-08-26T13:49:05.100 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ error_code=124 2023-08-26T13:49:05.101 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ set -e 2023-08-26T13:49:05.101 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ '[' 124 -eq 124 ']' 2023-08-26T13:49:05.101 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ return 0 2023-08-26T13:49:05.101 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-08-26T13:49:05.101 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ image=image_39 2023-08-26T13:49:05.103 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ create_image_and_enable_mirror cluster2 mirror image_39 snapshot 128M 2023-08-26T13:49:05.104 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster2 2023-08-26T13:49:05.104 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ shift 2023-08-26T13:49:05.104 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T13:49:05.104 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ shift 2023-08-26T13:49:05.105 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=image_39 2023-08-26T13:49:05.105 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ shift 2023-08-26T13:49:05.105 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local mode=snapshot 2023-08-26T13:49:05.105 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ '[' -n snapshot ']' 2023-08-26T13:49:05.106 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ shift 2023-08-26T13:49:05.106 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ create_image cluster2 mirror image_39 128M 2023-08-26T13:49:05.106 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster2 2023-08-26T13:49:05.106 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ shift 2023-08-26T13:49:05.106 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T13:49:05.107 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ shift 2023-08-26T13:49:05.107 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=image_39 2023-08-26T13:49:05.107 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ shift 2023-08-26T13:49:05.107 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local size=128 2023-08-26T13:49:05.108 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ '[' -n 128M ']' 2023-08-26T13:49:05.108 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ size=128M 2023-08-26T13:49:05.108 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ shift 2023-08-26T13:49:05.108 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock mirror/image_39 2023-08-26T13:49:05.147 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ '[' image = image ']' 2023-08-26T13:49:05.148 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ enable_mirror cluster2 mirror image_39 snapshot 2023-08-26T13:49:05.148 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster2 2023-08-26T13:49:05.148 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T13:49:05.148 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=image_39 2023-08-26T13:49:05.149 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local mode=snapshot 2023-08-26T13:49:05.149 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_39 snapshot 2023-08-26T13:49:06.058 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:Mirroring enabled 2023-08-26T13:49:06.066 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rbd --cluster=cluster2 info mirror/image_39 2023-08-26T13:49:06.103 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:rbd image 'image_39': 2023-08-26T13:49:06.103 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: size 128 MiB in 32 objects 2023-08-26T13:49:06.104 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: order 22 (4 MiB objects) 2023-08-26T13:49:06.104 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: snapshot_count: 1 2023-08-26T13:49:06.104 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: id: 13f9b855bbcf 2023-08-26T13:49:06.104 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: block_name_prefix: rbd_data.13f9b855bbcf 2023-08-26T13:49:06.104 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: format: 2 2023-08-26T13:49:06.104 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: features: layering, exclusive-lock 2023-08-26T13:49:06.105 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: op_features: 2023-08-26T13:49:06.105 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: flags: 2023-08-26T13:49:06.105 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: create_timestamp: Sat Aug 26 13:49:05 2023 2023-08-26T13:49:06.105 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: access_timestamp: Sat Aug 26 13:49:05 2023 2023-08-26T13:49:06.105 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: modify_timestamp: Sat Aug 26 13:49:05 2023 2023-08-26T13:49:06.106 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: mirroring state: enabled 2023-08-26T13:49:06.106 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: mirroring mode: snapshot 2023-08-26T13:49:06.106 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: mirroring global id: e777bb16-afba-4c52-994a-6980c13f1760 2023-08-26T13:49:06.106 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: mirroring primary: true 2023-08-26T13:49:06.107 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ '[' -n '' ']' 2023-08-26T13:49:06.107 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ stress_write_image cluster2 mirror image_39 2023-08-26T13:49:06.107 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster2 2023-08-26T13:49:06.108 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T13:49:06.108 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=image_39 2023-08-26T13:49:06.108 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2023-08-26T13:49:06.109 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local duration=13 2023-08-26T13:49:06.109 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ set +e 2023-08-26T13:49:06.109 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ timeout 13s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_39 --debug-rbd=20 --debug-journaler=20 2023-08-26T13:49:06.379 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:100 IOs, 68524032 bytes 2023-08-26T13:49:06.550 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:200 IOs, 115177472 bytes 2023-08-26T13:49:06.788 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:300 IOs, 189434880 bytes 2023-08-26T13:49:06.935 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:400 IOs, 199444480 bytes 2023-08-26T13:49:07.093 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:500 IOs, 225868800 bytes 2023-08-26T13:49:07.249 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:600 IOs, 257597440 bytes 2023-08-26T13:49:07.366 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:700 IOs, 267453440 bytes 2023-08-26T13:49:07.541 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:800 IOs, 305285120 bytes 2023-08-26T13:49:07.676 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:900 IOs, 324471808 bytes 2023-08-26T13:49:07.867 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:1000 IOs, 361962496 bytes 2023-08-26T13:49:08.044 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:1100 IOs, 401057792 bytes 2023-08-26T13:49:08.263 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:1200 IOs, 455249920 bytes 2023-08-26T13:49:08.421 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:1300 IOs, 482473984 bytes 2023-08-26T13:49:08.607 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:1400 IOs, 532849664 bytes 2023-08-26T13:49:08.804 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:1500 IOs, 575275008 bytes 2023-08-26T13:49:08.988 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:1600 IOs, 627840000 bytes 2023-08-26T13:49:09.109 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:1700 IOs, 641233920 bytes 2023-08-26T13:49:09.294 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:1800 IOs, 675766272 bytes 2023-08-26T13:49:09.564 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:1900 IOs, 800044032 bytes 2023-08-26T13:49:09.880 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:2000 IOs, 876152832 bytes 2023-08-26T13:49:10.040 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:2100 IOs, 902129664 bytes 2023-08-26T13:49:10.291 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:2200 IOs, 976347136 bytes 2023-08-26T13:49:10.481 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:2300 IOs, 1025006592 bytes 2023-08-26T13:49:10.595 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:2400 IOs, 1025615872 bytes 2023-08-26T13:49:10.758 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:2500 IOs, 1057780736 bytes 2023-08-26T13:49:10.949 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:2600 IOs, 1096185856 bytes 2023-08-26T13:49:11.146 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:2700 IOs, 1148078080 bytes 2023-08-26T13:49:11.356 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:2800 IOs, 1198909440 bytes 2023-08-26T13:49:11.603 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:2900 IOs, 1290306560 bytes 2023-08-26T13:49:11.803 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:3000 IOs, 1315944448 bytes 2023-08-26T13:49:12.015 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:3100 IOs, 1364589568 bytes 2023-08-26T13:49:12.271 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:3200 IOs, 1442506752 bytes 2023-08-26T13:49:12.411 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:3300 IOs, 1465764864 bytes 2023-08-26T13:49:12.605 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:3400 IOs, 1507952640 bytes 2023-08-26T13:49:12.751 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:3500 IOs, 1527703552 bytes 2023-08-26T13:49:12.902 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:3600 IOs, 1543732224 bytes 2023-08-26T13:49:13.038 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:3700 IOs, 1561570304 bytes 2023-08-26T13:49:13.153 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:3800 IOs, 1564745728 bytes 2023-08-26T13:49:13.349 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:3900 IOs, 1611384832 bytes 2023-08-26T13:49:13.505 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:4000 IOs, 1636406272 bytes 2023-08-26T13:49:13.688 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:4100 IOs, 1677587456 bytes 2023-08-26T13:49:13.831 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:4200 IOs, 1692403712 bytes 2023-08-26T13:49:13.972 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:4300 IOs, 1728466944 bytes 2023-08-26T13:49:14.113 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:4400 IOs, 1747089408 bytes 2023-08-26T13:49:14.275 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:4500 IOs, 1759238144 bytes 2023-08-26T13:49:14.471 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:4600 IOs, 1810165760 bytes 2023-08-26T13:49:14.653 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:4700 IOs, 1841571840 bytes 2023-08-26T13:49:14.761 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:4800 IOs, 1843562496 bytes 2023-08-26T13:49:14.971 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:4900 IOs, 1897822208 bytes 2023-08-26T13:49:15.129 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:5000 IOs, 1926619136 bytes 2023-08-26T13:49:15.262 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:5100 IOs, 1937689600 bytes 2023-08-26T13:49:15.415 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:5200 IOs, 1961198592 bytes 2023-08-26T13:49:15.612 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:5300 IOs, 2005025792 bytes 2023-08-26T13:49:15.760 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:5400 IOs, 2027754496 bytes 2023-08-26T13:49:15.984 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:5500 IOs, 2097461248 bytes 2023-08-26T13:49:16.173 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:5600 IOs, 2131954688 bytes 2023-08-26T13:49:16.314 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:5700 IOs, 2147042304 bytes 2023-08-26T13:49:16.527 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:5800 IOs, 2200414208 bytes 2023-08-26T13:49:16.646 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:5900 IOs, 2209105920 bytes 2023-08-26T13:49:16.805 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:6000 IOs, 2234954752 bytes 2023-08-26T13:49:16.925 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:6100 IOs, 2243994624 bytes 2023-08-26T13:49:17.179 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:6200 IOs, 2315505664 bytes 2023-08-26T13:49:17.360 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:6300 IOs, 2359666688 bytes 2023-08-26T13:49:17.588 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:6400 IOs, 2422622208 bytes 2023-08-26T13:49:17.822 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:6500 IOs, 2469022720 bytes 2023-08-26T13:49:18.029 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:6600 IOs, 2512030720 bytes 2023-08-26T13:49:18.182 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:6700 IOs, 2535019520 bytes 2023-08-26T13:49:18.371 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:6800 IOs, 2575528960 bytes 2023-08-26T13:49:18.524 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:6900 IOs, 2618746880 bytes 2023-08-26T13:49:18.892 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:7000 IOs, 2731134976 bytes 2023-08-26T13:49:19.114 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ error_code=124 2023-08-26T13:49:19.114 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ set -e 2023-08-26T13:49:19.115 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ '[' 124 -eq 124 ']' 2023-08-26T13:49:19.115 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ return 0 2023-08-26T13:49:19.115 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-08-26T13:49:19.115 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ image=image_40 2023-08-26T13:49:19.115 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ create_image_and_enable_mirror cluster2 mirror image_40 snapshot 128M 2023-08-26T13:49:19.116 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster2 2023-08-26T13:49:19.116 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ shift 2023-08-26T13:49:19.116 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T13:49:19.116 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ shift 2023-08-26T13:49:19.116 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=image_40 2023-08-26T13:49:19.117 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ shift 2023-08-26T13:49:19.117 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local mode=snapshot 2023-08-26T13:49:19.117 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ '[' -n snapshot ']' 2023-08-26T13:49:19.117 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ shift 2023-08-26T13:49:19.118 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ create_image cluster2 mirror image_40 128M 2023-08-26T13:49:19.118 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster2 2023-08-26T13:49:19.118 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ shift 2023-08-26T13:49:19.118 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T13:49:19.118 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ shift 2023-08-26T13:49:19.119 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=image_40 2023-08-26T13:49:19.119 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ shift 2023-08-26T13:49:19.119 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local size=128 2023-08-26T13:49:19.119 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ '[' -n 128M ']' 2023-08-26T13:49:19.119 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ size=128M 2023-08-26T13:49:19.120 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ shift 2023-08-26T13:49:19.120 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock mirror/image_40 2023-08-26T13:49:19.161 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ '[' image = image ']' 2023-08-26T13:49:19.161 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ enable_mirror cluster2 mirror image_40 snapshot 2023-08-26T13:49:19.161 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster2 2023-08-26T13:49:19.161 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T13:49:19.161 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=image_40 2023-08-26T13:49:19.162 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local mode=snapshot 2023-08-26T13:49:19.162 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_40 snapshot 2023-08-26T13:49:20.190 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:Mirroring enabled 2023-08-26T13:49:20.200 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rbd --cluster=cluster2 info mirror/image_40 2023-08-26T13:49:20.238 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:rbd image 'image_40': 2023-08-26T13:49:20.239 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: size 128 MiB in 32 objects 2023-08-26T13:49:20.239 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: order 22 (4 MiB objects) 2023-08-26T13:49:20.239 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: snapshot_count: 1 2023-08-26T13:49:20.239 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: id: 14025eadce4e 2023-08-26T13:49:20.240 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: block_name_prefix: rbd_data.14025eadce4e 2023-08-26T13:49:20.240 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: format: 2 2023-08-26T13:49:20.240 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: features: layering, exclusive-lock 2023-08-26T13:49:20.240 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: op_features: 2023-08-26T13:49:20.240 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: flags: 2023-08-26T13:49:20.241 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: create_timestamp: Sat Aug 26 13:49:19 2023 2023-08-26T13:49:20.241 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: access_timestamp: Sat Aug 26 13:49:19 2023 2023-08-26T13:49:20.241 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: modify_timestamp: Sat Aug 26 13:49:19 2023 2023-08-26T13:49:20.241 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: mirroring state: enabled 2023-08-26T13:49:20.241 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: mirroring mode: snapshot 2023-08-26T13:49:20.241 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: mirroring global id: 5bb319f8-2f9d-4a84-8756-0c66e011e111 2023-08-26T13:49:20.242 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: mirroring primary: true 2023-08-26T13:49:20.242 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ '[' -n '' ']' 2023-08-26T13:49:20.243 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ stress_write_image cluster2 mirror image_40 2023-08-26T13:49:20.243 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster2 2023-08-26T13:49:20.243 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T13:49:20.243 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=image_40 2023-08-26T13:49:20.244 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2023-08-26T13:49:20.244 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local duration=9 2023-08-26T13:49:20.244 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ set +e 2023-08-26T13:49:20.245 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ timeout 9s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_40 --debug-rbd=20 --debug-journaler=20 2023-08-26T13:49:20.383 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:100 IOs, 625664 bytes 2023-08-26T13:49:20.516 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:200 IOs, 14612480 bytes 2023-08-26T13:49:20.670 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:300 IOs, 43498496 bytes 2023-08-26T13:49:20.825 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:400 IOs, 83585024 bytes 2023-08-26T13:49:21.020 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:500 IOs, 120520704 bytes 2023-08-26T13:49:21.164 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:600 IOs, 140571648 bytes 2023-08-26T13:49:21.348 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:700 IOs, 178966528 bytes 2023-08-26T13:49:21.524 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:800 IOs, 211519488 bytes 2023-08-26T13:49:21.680 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:900 IOs, 235264000 bytes 2023-08-26T13:49:21.948 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:1000 IOs, 314987520 bytes 2023-08-26T13:49:22.203 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:1100 IOs, 370899968 bytes 2023-08-26T13:49:22.397 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:1200 IOs, 407333888 bytes 2023-08-26T13:49:22.616 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:1300 IOs, 460627968 bytes 2023-08-26T13:49:22.780 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:1400 IOs, 483915776 bytes 2023-08-26T13:49:22.956 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:1500 IOs, 523299840 bytes 2023-08-26T13:49:23.114 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:1600 IOs, 550587392 bytes 2023-08-26T13:49:23.342 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:1700 IOs, 558391296 bytes 2023-08-26T13:49:23.484 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:1800 IOs, 581585920 bytes 2023-08-26T13:49:23.722 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.0 with skip (0.6365198360003308) > weight (0.25) 2023-08-26T13:49:23.723 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.1 with skip (0.546023229351045) > weight (0.25) 2023-08-26T13:49:23.723 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.2 with skip (0.6101635828833709) > weight (0.25) 2023-08-26T13:49:23.723 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.3 with skip (0.8318807591542019) > weight (0.25) 2023-08-26T13:49:23.724 INFO:tasks.rbd_mirror_thrash:waiting for 101 secs before thrashing 2023-08-26T13:49:23.772 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:1900 IOs, 680267776 bytes 2023-08-26T13:49:24.032 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:2000 IOs, 736811008 bytes 2023-08-26T13:49:24.152 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:2100 IOs, 737426432 bytes 2023-08-26T13:49:24.347 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:2200 IOs, 777155584 bytes 2023-08-26T13:49:24.484 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:2300 IOs, 791472128 bytes 2023-08-26T13:49:24.686 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:2400 IOs, 833932288 bytes 2023-08-26T13:49:24.824 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:2500 IOs, 864908288 bytes 2023-08-26T13:49:25.035 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:2600 IOs, 900410368 bytes 2023-08-26T13:49:25.243 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:2700 IOs, 952513536 bytes 2023-08-26T13:49:25.445 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:2800 IOs, 1006642176 bytes 2023-08-26T13:49:25.611 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:2900 IOs, 1038322688 bytes 2023-08-26T13:49:25.909 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:3000 IOs, 1126132736 bytes 2023-08-26T13:49:26.096 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:3100 IOs, 1183375360 bytes 2023-08-26T13:49:26.336 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:3200 IOs, 1236545536 bytes 2023-08-26T13:49:26.513 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:3300 IOs, 1276892160 bytes 2023-08-26T13:49:26.741 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:3400 IOs, 1347706880 bytes 2023-08-26T13:49:26.988 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:3500 IOs, 1423253504 bytes 2023-08-26T13:49:27.182 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:3600 IOs, 1471676416 bytes 2023-08-26T13:49:27.344 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:3700 IOs, 1506177024 bytes 2023-08-26T13:49:27.472 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:3800 IOs, 1515249664 bytes 2023-08-26T13:49:27.629 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:3900 IOs, 1547890688 bytes 2023-08-26T13:49:27.814 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:4000 IOs, 1593886720 bytes 2023-08-26T13:49:27.922 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:4100 IOs, 1596274688 bytes 2023-08-26T13:49:28.019 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:4200 IOs, 1596886016 bytes 2023-08-26T13:49:28.182 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:4300 IOs, 1636442112 bytes 2023-08-26T13:49:28.380 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:4400 IOs, 1678732288 bytes 2023-08-26T13:49:28.512 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:4500 IOs, 1691873280 bytes 2023-08-26T13:49:28.632 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:4600 IOs, 1692486656 bytes 2023-08-26T13:49:28.757 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:4700 IOs, 1703388160 bytes 2023-08-26T13:49:28.993 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:4800 IOs, 1762819072 bytes 2023-08-26T13:49:29.163 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:4900 IOs, 1795547136 bytes 2023-08-26T13:49:29.250 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ error_code=124 2023-08-26T13:49:29.251 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ set -e 2023-08-26T13:49:29.251 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ '[' 124 -eq 124 ']' 2023-08-26T13:49:29.251 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ return 0 2023-08-26T13:49:29.251 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-08-26T13:49:29.251 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ image=image_41 2023-08-26T13:49:29.252 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ create_image_and_enable_mirror cluster2 mirror image_41 snapshot 128M 2023-08-26T13:49:29.252 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster2 2023-08-26T13:49:29.252 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ shift 2023-08-26T13:49:29.252 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T13:49:29.252 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ shift 2023-08-26T13:49:29.253 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=image_41 2023-08-26T13:49:29.253 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ shift 2023-08-26T13:49:29.253 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local mode=snapshot 2023-08-26T13:49:29.253 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ '[' -n snapshot ']' 2023-08-26T13:49:29.253 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ shift 2023-08-26T13:49:29.254 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ create_image cluster2 mirror image_41 128M 2023-08-26T13:49:29.254 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster2 2023-08-26T13:49:29.254 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ shift 2023-08-26T13:49:29.254 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T13:49:29.254 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ shift 2023-08-26T13:49:29.255 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=image_41 2023-08-26T13:49:29.255 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ shift 2023-08-26T13:49:29.255 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local size=128 2023-08-26T13:49:29.255 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ '[' -n 128M ']' 2023-08-26T13:49:29.255 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ size=128M 2023-08-26T13:49:29.255 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ shift 2023-08-26T13:49:29.256 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock mirror/image_41 2023-08-26T13:49:29.295 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ '[' image = image ']' 2023-08-26T13:49:29.296 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ enable_mirror cluster2 mirror image_41 snapshot 2023-08-26T13:49:29.296 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster2 2023-08-26T13:49:29.296 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T13:49:29.296 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=image_41 2023-08-26T13:49:29.296 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local mode=snapshot 2023-08-26T13:49:29.296 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_41 snapshot 2023-08-26T13:49:30.171 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:Mirroring enabled 2023-08-26T13:49:30.179 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rbd --cluster=cluster2 info mirror/image_41 2023-08-26T13:49:30.215 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:rbd image 'image_41': 2023-08-26T13:49:30.216 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: size 128 MiB in 32 objects 2023-08-26T13:49:30.216 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: order 22 (4 MiB objects) 2023-08-26T13:49:30.216 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: snapshot_count: 1 2023-08-26T13:49:30.216 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: id: 140b1de71f76 2023-08-26T13:49:30.216 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: block_name_prefix: rbd_data.140b1de71f76 2023-08-26T13:49:30.217 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: format: 2 2023-08-26T13:49:30.217 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: features: layering, exclusive-lock 2023-08-26T13:49:30.217 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: op_features: 2023-08-26T13:49:30.217 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: flags: 2023-08-26T13:49:30.217 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: create_timestamp: Sat Aug 26 13:49:29 2023 2023-08-26T13:49:30.217 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: access_timestamp: Sat Aug 26 13:49:29 2023 2023-08-26T13:49:30.218 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: modify_timestamp: Sat Aug 26 13:49:29 2023 2023-08-26T13:49:30.218 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: mirroring state: enabled 2023-08-26T13:49:30.218 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: mirroring mode: snapshot 2023-08-26T13:49:30.218 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: mirroring global id: bfd70442-dd63-468e-9119-77e3357316c5 2023-08-26T13:49:30.219 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: mirroring primary: true 2023-08-26T13:49:30.220 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ '[' -n '' ']' 2023-08-26T13:49:30.220 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ stress_write_image cluster2 mirror image_41 2023-08-26T13:49:30.220 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster2 2023-08-26T13:49:30.221 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T13:49:30.221 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=image_41 2023-08-26T13:49:30.221 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2023-08-26T13:49:30.221 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local duration=12 2023-08-26T13:49:30.221 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ set +e 2023-08-26T13:49:30.222 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ timeout 12s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_41 --debug-rbd=20 --debug-journaler=20 2023-08-26T13:49:30.377 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:100 IOs, 8006656 bytes 2023-08-26T13:49:30.559 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:200 IOs, 47429632 bytes 2023-08-26T13:49:30.719 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:300 IOs, 68934656 bytes 2023-08-26T13:49:30.853 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:400 IOs, 90766336 bytes 2023-08-26T13:49:31.073 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:500 IOs, 139170816 bytes 2023-08-26T13:49:31.266 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:600 IOs, 179657728 bytes 2023-08-26T13:49:31.429 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:700 IOs, 208888832 bytes 2023-08-26T13:49:31.605 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:800 IOs, 243431424 bytes 2023-08-26T13:49:31.754 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:900 IOs, 266094592 bytes 2023-08-26T13:49:31.925 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:1000 IOs, 327183360 bytes 2023-08-26T13:49:32.152 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:1100 IOs, 372390912 bytes 2023-08-26T13:49:32.354 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:1200 IOs, 402461696 bytes 2023-08-26T13:49:32.515 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:1300 IOs, 428103680 bytes 2023-08-26T13:49:32.776 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:1400 IOs, 503307264 bytes 2023-08-26T13:49:32.969 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:1500 IOs, 544249856 bytes 2023-08-26T13:49:33.144 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:1600 IOs, 587460608 bytes 2023-08-26T13:49:33.342 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:1700 IOs, 617694208 bytes 2023-08-26T13:49:33.538 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:1800 IOs, 662618112 bytes 2023-08-26T13:49:33.720 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:1900 IOs, 699348992 bytes 2023-08-26T13:49:33.853 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:2000 IOs, 707814400 bytes 2023-08-26T13:49:34.109 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:2100 IOs, 781992960 bytes 2023-08-26T13:49:34.252 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:2200 IOs, 804302848 bytes 2023-08-26T13:49:34.437 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:2300 IOs, 844087296 bytes 2023-08-26T13:49:34.700 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:2400 IOs, 922409984 bytes 2023-08-26T13:49:34.842 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:2500 IOs, 942800896 bytes 2023-08-26T13:49:35.030 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:2600 IOs, 988707840 bytes 2023-08-26T13:49:35.140 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:2700 IOs, 989331456 bytes 2023-08-26T13:49:35.318 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:2800 IOs, 1026282496 bytes 2023-08-26T13:49:35.468 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:2900 IOs, 1055616000 bytes 2023-08-26T13:49:35.707 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:3000 IOs, 1115718656 bytes 2023-08-26T13:49:35.995 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:3100 IOs, 1203510272 bytes 2023-08-26T13:49:36.204 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:3200 IOs, 1256084480 bytes 2023-08-26T13:49:36.393 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:3300 IOs, 1299819520 bytes 2023-08-26T13:49:36.509 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.0 with skip (0.49035668199010785) > weight (0.25) 2023-08-26T13:49:36.510 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.1 with skip (0.4384803571641527) > weight (0.25) 2023-08-26T13:49:36.510 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.2 with skip (0.6314907288726752) > weight (0.25) 2023-08-26T13:49:36.510 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.3 with skip (0.6084417616868847) > weight (0.25) 2023-08-26T13:49:36.511 INFO:tasks.rbd_mirror_thrash:waiting for 83 secs before thrashing 2023-08-26T13:49:36.556 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:3400 IOs, 1328960512 bytes 2023-08-26T13:49:36.782 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:3500 IOs, 1385802752 bytes 2023-08-26T13:49:36.927 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:3600 IOs, 1406637056 bytes 2023-08-26T13:49:37.040 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:3700 IOs, 1411964928 bytes 2023-08-26T13:49:37.231 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:3800 IOs, 1447731200 bytes 2023-08-26T13:49:37.455 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:3900 IOs, 1504257024 bytes 2023-08-26T13:49:37.646 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:4000 IOs, 1544988672 bytes 2023-08-26T13:49:37.806 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:4100 IOs, 1577118720 bytes 2023-08-26T13:49:38.010 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:4200 IOs, 1630618624 bytes 2023-08-26T13:49:38.281 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:4300 IOs, 1720200192 bytes 2023-08-26T13:49:38.418 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:4400 IOs, 1735408640 bytes 2023-08-26T13:49:38.641 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:4500 IOs, 1799482368 bytes 2023-08-26T13:49:38.833 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:4600 IOs, 1846252544 bytes 2023-08-26T13:49:38.935 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:4700 IOs, 1846873088 bytes 2023-08-26T13:49:39.079 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:4800 IOs, 1867154432 bytes 2023-08-26T13:49:39.225 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:4900 IOs, 1892343808 bytes 2023-08-26T13:49:39.386 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:5000 IOs, 1928146944 bytes 2023-08-26T13:49:39.509 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:5100 IOs, 1929449472 bytes 2023-08-26T13:49:39.668 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:5200 IOs, 1951705088 bytes 2023-08-26T13:49:39.890 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:5300 IOs, 2019926016 bytes 2023-08-26T13:49:40.085 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:5400 IOs, 2063522816 bytes 2023-08-26T13:49:40.186 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:5500 IOs, 2064115712 bytes 2023-08-26T13:49:40.310 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:5600 IOs, 2073724928 bytes 2023-08-26T13:49:40.515 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:5700 IOs, 2150462464 bytes 2023-08-26T13:49:40.757 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:5800 IOs, 2190402560 bytes 2023-08-26T13:49:40.956 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:5900 IOs, 2234946560 bytes 2023-08-26T13:49:41.135 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:6000 IOs, 2278449152 bytes 2023-08-26T13:49:41.278 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:6100 IOs, 2296408064 bytes 2023-08-26T13:49:41.433 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:6200 IOs, 2335922176 bytes 2023-08-26T13:49:41.666 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:6300 IOs, 2400145408 bytes 2023-08-26T13:49:41.813 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:6400 IOs, 2423067648 bytes 2023-08-26T13:49:41.919 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:6500 IOs, 2423669760 bytes 2023-08-26T13:49:42.090 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:6600 IOs, 2458414080 bytes 2023-08-26T13:49:42.194 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:6700 IOs, 2459026432 bytes 2023-08-26T13:49:42.226 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ error_code=124 2023-08-26T13:49:42.227 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ set -e 2023-08-26T13:49:42.227 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ '[' 124 -eq 124 ']' 2023-08-26T13:49:42.228 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ return 0 2023-08-26T13:49:42.228 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-08-26T13:49:42.228 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ image=image_42 2023-08-26T13:49:42.229 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ create_image_and_enable_mirror cluster2 mirror image_42 snapshot 128M 2023-08-26T13:49:42.229 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster2 2023-08-26T13:49:42.229 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ shift 2023-08-26T13:49:42.230 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T13:49:42.230 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ shift 2023-08-26T13:49:42.230 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=image_42 2023-08-26T13:49:42.231 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ shift 2023-08-26T13:49:42.231 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local mode=snapshot 2023-08-26T13:49:42.231 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ '[' -n snapshot ']' 2023-08-26T13:49:42.232 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ shift 2023-08-26T13:49:42.232 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ create_image cluster2 mirror image_42 128M 2023-08-26T13:49:42.232 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster2 2023-08-26T13:49:42.232 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ shift 2023-08-26T13:49:42.233 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T13:49:42.233 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ shift 2023-08-26T13:49:42.233 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=image_42 2023-08-26T13:49:42.233 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ shift 2023-08-26T13:49:42.234 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local size=128 2023-08-26T13:49:42.234 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ '[' -n 128M ']' 2023-08-26T13:49:42.234 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ size=128M 2023-08-26T13:49:42.235 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ shift 2023-08-26T13:49:42.235 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock mirror/image_42 2023-08-26T13:49:42.272 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ '[' image = image ']' 2023-08-26T13:49:42.273 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ enable_mirror cluster2 mirror image_42 snapshot 2023-08-26T13:49:42.273 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster2 2023-08-26T13:49:42.274 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T13:49:42.274 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=image_42 2023-08-26T13:49:42.274 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local mode=snapshot 2023-08-26T13:49:42.275 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_42 snapshot 2023-08-26T13:49:43.215 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:Mirroring enabled 2023-08-26T13:49:43.223 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rbd --cluster=cluster2 info mirror/image_42 2023-08-26T13:49:43.261 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:rbd image 'image_42': 2023-08-26T13:49:43.261 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: size 128 MiB in 32 objects 2023-08-26T13:49:43.261 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: order 22 (4 MiB objects) 2023-08-26T13:49:43.262 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: snapshot_count: 1 2023-08-26T13:49:43.262 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: id: 1413aa18ce6e 2023-08-26T13:49:43.262 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: block_name_prefix: rbd_data.1413aa18ce6e 2023-08-26T13:49:43.262 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: format: 2 2023-08-26T13:49:43.262 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: features: layering, exclusive-lock 2023-08-26T13:49:43.263 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: op_features: 2023-08-26T13:49:43.263 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: flags: 2023-08-26T13:49:43.263 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: create_timestamp: Sat Aug 26 13:49:42 2023 2023-08-26T13:49:43.263 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: access_timestamp: Sat Aug 26 13:49:42 2023 2023-08-26T13:49:43.263 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: modify_timestamp: Sat Aug 26 13:49:42 2023 2023-08-26T13:49:43.263 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: mirroring state: enabled 2023-08-26T13:49:43.264 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: mirroring mode: snapshot 2023-08-26T13:49:43.264 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: mirroring global id: 60b957ec-f068-4fc2-9c4b-457a79a9a72f 2023-08-26T13:49:43.264 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: mirroring primary: true 2023-08-26T13:49:43.265 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ '[' -n '' ']' 2023-08-26T13:49:43.265 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ stress_write_image cluster2 mirror image_42 2023-08-26T13:49:43.265 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster2 2023-08-26T13:49:43.266 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T13:49:43.266 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=image_42 2023-08-26T13:49:43.266 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2023-08-26T13:49:43.267 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local duration=6 2023-08-26T13:49:43.267 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ set +e 2023-08-26T13:49:43.267 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ timeout 6s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_42 --debug-rbd=20 --debug-journaler=20 2023-08-26T13:49:43.510 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:100 IOs, 48705536 bytes 2023-08-26T13:49:43.681 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:200 IOs, 92406784 bytes 2023-08-26T13:49:43.864 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:300 IOs, 128171008 bytes 2023-08-26T13:49:44.032 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:400 IOs, 158156800 bytes 2023-08-26T13:49:44.190 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:500 IOs, 186634240 bytes 2023-08-26T13:49:44.392 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:600 IOs, 237137920 bytes 2023-08-26T13:49:44.590 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:700 IOs, 286128128 bytes 2023-08-26T13:49:44.765 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:800 IOs, 318720000 bytes 2023-08-26T13:49:44.905 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:900 IOs, 336183296 bytes 2023-08-26T13:49:45.140 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:1000 IOs, 399803392 bytes 2023-08-26T13:49:45.274 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:1100 IOs, 414761984 bytes 2023-08-26T13:49:45.473 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:1200 IOs, 451215360 bytes 2023-08-26T13:49:45.681 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:1300 IOs, 495026176 bytes 2023-08-26T13:49:45.897 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:1400 IOs, 544533504 bytes 2023-08-26T13:49:46.049 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:1500 IOs, 566051840 bytes 2023-08-26T13:49:46.279 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:1600 IOs, 626625536 bytes 2023-08-26T13:49:46.405 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:1700 IOs, 640093184 bytes 2023-08-26T13:49:46.517 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:1800 IOs, 641872896 bytes 2023-08-26T13:49:46.657 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:1900 IOs, 686406656 bytes 2023-08-26T13:49:46.886 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:2000 IOs, 725735424 bytes 2023-08-26T13:49:47.085 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:2100 IOs, 768084992 bytes 2023-08-26T13:49:47.240 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:2200 IOs, 793284608 bytes 2023-08-26T13:49:47.372 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:2300 IOs, 804421632 bytes 2023-08-26T13:49:47.530 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:2400 IOs, 824940544 bytes 2023-08-26T13:49:47.654 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:2500 IOs, 829015040 bytes 2023-08-26T13:49:47.795 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:2600 IOs, 846223360 bytes 2023-08-26T13:49:47.960 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:2700 IOs, 877917184 bytes 2023-08-26T13:49:48.259 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:2800 IOs, 975600640 bytes 2023-08-26T13:49:48.504 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:2900 IOs, 1038142464 bytes 2023-08-26T13:49:48.695 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:3000 IOs, 1078776832 bytes 2023-08-26T13:49:48.991 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:3100 IOs, 1173715968 bytes 2023-08-26T13:49:49.150 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:3200 IOs, 1206550528 bytes 2023-08-26T13:49:49.279 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ error_code=124 2023-08-26T13:49:49.279 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ set -e 2023-08-26T13:49:49.280 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ '[' 124 -eq 124 ']' 2023-08-26T13:49:49.280 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ return 0 2023-08-26T13:49:49.280 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-08-26T13:49:49.280 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ image=image_43 2023-08-26T13:49:49.281 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ create_image_and_enable_mirror cluster2 mirror image_43 snapshot 128M 2023-08-26T13:49:49.281 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster2 2023-08-26T13:49:49.281 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ shift 2023-08-26T13:49:49.281 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T13:49:49.282 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ shift 2023-08-26T13:49:49.282 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=image_43 2023-08-26T13:49:49.282 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ shift 2023-08-26T13:49:49.282 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local mode=snapshot 2023-08-26T13:49:49.282 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ '[' -n snapshot ']' 2023-08-26T13:49:49.283 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ shift 2023-08-26T13:49:49.283 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ create_image cluster2 mirror image_43 128M 2023-08-26T13:49:49.283 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster2 2023-08-26T13:49:49.283 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ shift 2023-08-26T13:49:49.283 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T13:49:49.284 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ shift 2023-08-26T13:49:49.284 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=image_43 2023-08-26T13:49:49.284 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ shift 2023-08-26T13:49:49.284 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local size=128 2023-08-26T13:49:49.284 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ '[' -n 128M ']' 2023-08-26T13:49:49.284 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ size=128M 2023-08-26T13:49:49.285 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ shift 2023-08-26T13:49:49.285 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock mirror/image_43 2023-08-26T13:49:49.330 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ '[' image = image ']' 2023-08-26T13:49:49.330 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ enable_mirror cluster2 mirror image_43 snapshot 2023-08-26T13:49:49.331 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster2 2023-08-26T13:49:49.331 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T13:49:49.331 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=image_43 2023-08-26T13:49:49.331 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local mode=snapshot 2023-08-26T13:49:49.332 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_43 snapshot 2023-08-26T13:49:50.280 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:Mirroring enabled 2023-08-26T13:49:50.288 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rbd --cluster=cluster2 info mirror/image_43 2023-08-26T13:49:50.325 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:rbd image 'image_43': 2023-08-26T13:49:50.325 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: size 128 MiB in 32 objects 2023-08-26T13:49:50.325 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: order 22 (4 MiB objects) 2023-08-26T13:49:50.326 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: snapshot_count: 1 2023-08-26T13:49:50.326 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: id: 141c29bcdd48 2023-08-26T13:49:50.326 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: block_name_prefix: rbd_data.141c29bcdd48 2023-08-26T13:49:50.326 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: format: 2 2023-08-26T13:49:50.326 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: features: layering, exclusive-lock 2023-08-26T13:49:50.327 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: op_features: 2023-08-26T13:49:50.327 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: flags: 2023-08-26T13:49:50.327 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: create_timestamp: Sat Aug 26 13:49:49 2023 2023-08-26T13:49:50.327 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: access_timestamp: Sat Aug 26 13:49:49 2023 2023-08-26T13:49:50.327 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: modify_timestamp: Sat Aug 26 13:49:49 2023 2023-08-26T13:49:50.328 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: mirroring state: enabled 2023-08-26T13:49:50.328 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: mirroring mode: snapshot 2023-08-26T13:49:50.328 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: mirroring global id: 93289c2f-b179-4509-b3aa-d0fd920c15e7 2023-08-26T13:49:50.328 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: mirroring primary: true 2023-08-26T13:49:50.329 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ '[' -n '' ']' 2023-08-26T13:49:50.329 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ stress_write_image cluster2 mirror image_43 2023-08-26T13:49:50.329 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster2 2023-08-26T13:49:50.330 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T13:49:50.330 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=image_43 2023-08-26T13:49:50.330 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2023-08-26T13:49:50.331 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local duration=14 2023-08-26T13:49:50.331 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ set +e 2023-08-26T13:49:50.331 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ timeout 14s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_43 --debug-rbd=20 --debug-journaler=20 2023-08-26T13:49:50.607 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:100 IOs, 77764608 bytes 2023-08-26T13:49:50.802 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:200 IOs, 111188992 bytes 2023-08-26T13:49:51.063 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:300 IOs, 187097088 bytes 2023-08-26T13:49:51.239 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:400 IOs, 221244416 bytes 2023-08-26T13:49:51.672 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:500 IOs, 378280960 bytes 2023-08-26T13:49:51.806 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:600 IOs, 408725504 bytes 2023-08-26T13:49:52.038 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:700 IOs, 462278656 bytes 2023-08-26T13:49:52.223 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:800 IOs, 502863872 bytes 2023-08-26T13:49:52.514 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:900 IOs, 591545344 bytes 2023-08-26T13:49:52.825 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:1000 IOs, 685683712 bytes 2023-08-26T13:49:52.951 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:1100 IOs, 688136192 bytes 2023-08-26T13:49:53.176 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:1200 IOs, 747528192 bytes 2023-08-26T13:49:53.447 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:1300 IOs, 821639168 bytes 2023-08-26T13:49:53.599 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:1400 IOs, 830656512 bytes 2023-08-26T13:49:53.864 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:1500 IOs, 879512576 bytes 2023-08-26T13:49:54.008 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:1600 IOs, 893721600 bytes 2023-08-26T13:49:54.171 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:1700 IOs, 927566848 bytes 2023-08-26T13:49:54.443 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:1800 IOs, 986517504 bytes 2023-08-26T13:49:54.608 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:1900 IOs, 1009245184 bytes 2023-08-26T13:49:54.751 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:2000 IOs, 1024926720 bytes 2023-08-26T13:49:54.864 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:2100 IOs, 1031790592 bytes 2023-08-26T13:49:54.978 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:2200 IOs, 1032432640 bytes 2023-08-26T13:49:55.149 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:2300 IOs, 1062107136 bytes 2023-08-26T13:49:55.344 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:2400 IOs, 1111105536 bytes 2023-08-26T13:49:55.502 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:2500 IOs, 1148076032 bytes 2023-08-26T13:49:55.654 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:2600 IOs, 1161169920 bytes 2023-08-26T13:49:55.857 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:2700 IOs, 1213066240 bytes 2023-08-26T13:49:55.989 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:2800 IOs, 1223320576 bytes 2023-08-26T13:49:56.120 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:2900 IOs, 1234021376 bytes 2023-08-26T13:49:56.291 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:3000 IOs, 1259855872 bytes 2023-08-26T13:49:56.585 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:3100 IOs, 1350716416 bytes 2023-08-26T13:49:56.796 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:3200 IOs, 1407808512 bytes 2023-08-26T13:49:56.953 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:3300 IOs, 1434677248 bytes 2023-08-26T13:49:57.191 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:3400 IOs, 1527152640 bytes 2023-08-26T13:49:57.382 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:3500 IOs, 1553988608 bytes 2023-08-26T13:49:57.595 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:3600 IOs, 1602814976 bytes 2023-08-26T13:49:57.766 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:3700 IOs, 1636643840 bytes 2023-08-26T13:49:58.118 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:3800 IOs, 1766165504 bytes 2023-08-26T13:49:58.289 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:3900 IOs, 1798962176 bytes 2023-08-26T13:49:58.458 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:4000 IOs, 1821805568 bytes 2023-08-26T13:49:58.583 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:4100 IOs, 1831732224 bytes 2023-08-26T13:49:58.703 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:4200 IOs, 1834648576 bytes 2023-08-26T13:49:58.951 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:4300 IOs, 1912069120 bytes 2023-08-26T13:49:59.189 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:4400 IOs, 1971086336 bytes 2023-08-26T13:49:59.295 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:4500 IOs, 1971686400 bytes 2023-08-26T13:49:59.431 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:4600 IOs, 1985370112 bytes 2023-08-26T13:49:59.579 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:4700 IOs, 2010661888 bytes 2023-08-26T13:49:59.757 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:4800 IOs, 2038814720 bytes 2023-08-26T13:49:59.901 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:4900 IOs, 2077998080 bytes 2023-08-26T13:50:00.092 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:5000 IOs, 2105731072 bytes 2023-08-26T13:50:00.270 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:5100 IOs, 2144480256 bytes 2023-08-26T13:50:00.622 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:5200 IOs, 2228220928 bytes 2023-08-26T13:50:01.048 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:5300 IOs, 2285810688 bytes 2023-08-26T13:50:01.269 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:5400 IOs, 2335768576 bytes 2023-08-26T13:50:01.401 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:5500 IOs, 2342035456 bytes 2023-08-26T13:50:01.644 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:5600 IOs, 2427123712 bytes 2023-08-26T13:50:01.941 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:5700 IOs, 2505633792 bytes 2023-08-26T13:50:02.103 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:5800 IOs, 2544890880 bytes 2023-08-26T13:50:02.313 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:5900 IOs, 2591621120 bytes 2023-08-26T13:50:02.443 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:6000 IOs, 2595972096 bytes 2023-08-26T13:50:02.606 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:6100 IOs, 2619831296 bytes 2023-08-26T13:50:02.784 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:6200 IOs, 2657704960 bytes 2023-08-26T13:50:02.959 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:6300 IOs, 2677314560 bytes 2023-08-26T13:50:03.148 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:6400 IOs, 2724477952 bytes 2023-08-26T13:50:03.276 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:6500 IOs, 2739141632 bytes 2023-08-26T13:50:03.512 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:6600 IOs, 2813356032 bytes 2023-08-26T13:50:03.654 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:6700 IOs, 2825590784 bytes 2023-08-26T13:50:03.900 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:6800 IOs, 2886273024 bytes 2023-08-26T13:50:04.042 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:6900 IOs, 2903812096 bytes 2023-08-26T13:50:04.242 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:7000 IOs, 2949391360 bytes 2023-08-26T13:50:04.338 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ error_code=124 2023-08-26T13:50:04.338 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ set -e 2023-08-26T13:50:04.338 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ '[' 124 -eq 124 ']' 2023-08-26T13:50:04.339 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ return 0 2023-08-26T13:50:04.339 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-08-26T13:50:04.339 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ image=image_44 2023-08-26T13:50:04.339 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ create_image_and_enable_mirror cluster2 mirror image_44 snapshot 128M 2023-08-26T13:50:04.339 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster2 2023-08-26T13:50:04.340 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ shift 2023-08-26T13:50:04.340 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T13:50:04.340 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ shift 2023-08-26T13:50:04.340 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=image_44 2023-08-26T13:50:04.340 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ shift 2023-08-26T13:50:04.340 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local mode=snapshot 2023-08-26T13:50:04.341 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ '[' -n snapshot ']' 2023-08-26T13:50:04.341 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ shift 2023-08-26T13:50:04.341 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ create_image cluster2 mirror image_44 128M 2023-08-26T13:50:04.341 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster2 2023-08-26T13:50:04.341 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ shift 2023-08-26T13:50:04.342 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T13:50:04.342 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ shift 2023-08-26T13:50:04.342 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=image_44 2023-08-26T13:50:04.342 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ shift 2023-08-26T13:50:04.342 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local size=128 2023-08-26T13:50:04.343 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ '[' -n 128M ']' 2023-08-26T13:50:04.343 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ size=128M 2023-08-26T13:50:04.343 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ shift 2023-08-26T13:50:04.343 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock mirror/image_44 2023-08-26T13:50:04.385 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ '[' image = image ']' 2023-08-26T13:50:04.385 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ enable_mirror cluster2 mirror image_44 snapshot 2023-08-26T13:50:04.385 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster2 2023-08-26T13:50:04.385 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T13:50:04.386 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=image_44 2023-08-26T13:50:04.386 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local mode=snapshot 2023-08-26T13:50:04.386 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_44 snapshot 2023-08-26T13:50:05.087 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:Mirroring enabled 2023-08-26T13:50:05.098 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rbd --cluster=cluster2 info mirror/image_44 2023-08-26T13:50:05.133 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:rbd image 'image_44': 2023-08-26T13:50:05.134 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: size 128 MiB in 32 objects 2023-08-26T13:50:05.134 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: order 22 (4 MiB objects) 2023-08-26T13:50:05.134 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: snapshot_count: 1 2023-08-26T13:50:05.134 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: id: 14251148e5ae 2023-08-26T13:50:05.134 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: block_name_prefix: rbd_data.14251148e5ae 2023-08-26T13:50:05.135 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: format: 2 2023-08-26T13:50:05.135 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: features: layering, exclusive-lock 2023-08-26T13:50:05.135 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: op_features: 2023-08-26T13:50:05.135 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: flags: 2023-08-26T13:50:05.136 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: create_timestamp: Sat Aug 26 13:50:04 2023 2023-08-26T13:50:05.136 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: access_timestamp: Sat Aug 26 13:50:04 2023 2023-08-26T13:50:05.136 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: modify_timestamp: Sat Aug 26 13:50:04 2023 2023-08-26T13:50:05.136 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: mirroring state: enabled 2023-08-26T13:50:05.136 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: mirroring mode: snapshot 2023-08-26T13:50:05.136 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: mirroring global id: 11ee1779-6c0f-4f0f-8432-721939d3a164 2023-08-26T13:50:05.137 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: mirroring primary: true 2023-08-26T13:50:05.137 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ '[' -n '' ']' 2023-08-26T13:50:05.138 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ stress_write_image cluster2 mirror image_44 2023-08-26T13:50:05.138 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster2 2023-08-26T13:50:05.138 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T13:50:05.138 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=image_44 2023-08-26T13:50:05.138 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2023-08-26T13:50:05.139 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local duration=10 2023-08-26T13:50:05.139 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ set +e 2023-08-26T13:50:05.139 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ timeout 10s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_44 --debug-rbd=20 --debug-journaler=20 2023-08-26T13:50:05.434 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:100 IOs, 73676800 bytes 2023-08-26T13:50:05.684 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:200 IOs, 140531712 bytes 2023-08-26T13:50:05.817 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:300 IOs, 179508224 bytes 2023-08-26T13:50:06.003 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:400 IOs, 191759360 bytes 2023-08-26T13:50:06.175 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:500 IOs, 235369472 bytes 2023-08-26T13:50:06.325 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:600 IOs, 239330304 bytes 2023-08-26T13:50:06.553 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:700 IOs, 318356480 bytes 2023-08-26T13:50:06.778 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:800 IOs, 358670336 bytes 2023-08-26T13:50:06.880 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:900 IOs, 359292928 bytes 2023-08-26T13:50:07.135 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:1000 IOs, 428054528 bytes 2023-08-26T13:50:07.312 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:1100 IOs, 467592192 bytes 2023-08-26T13:50:07.465 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:1200 IOs, 502440960 bytes 2023-08-26T13:50:07.693 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:1300 IOs, 556113920 bytes 2023-08-26T13:50:07.952 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:1400 IOs, 631727104 bytes 2023-08-26T13:50:08.112 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:1500 IOs, 660827136 bytes 2023-08-26T13:50:08.272 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:1600 IOs, 689219584 bytes 2023-08-26T13:50:08.484 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:1700 IOs, 740465664 bytes 2023-08-26T13:50:08.682 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:1800 IOs, 785931264 bytes 2023-08-26T13:50:08.925 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:1900 IOs, 862854144 bytes 2023-08-26T13:50:09.169 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:2000 IOs, 913005568 bytes 2023-08-26T13:50:09.320 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:2100 IOs, 940747776 bytes 2023-08-26T13:50:09.486 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:2200 IOs, 965827584 bytes 2023-08-26T13:50:09.700 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:2300 IOs, 1030746112 bytes 2023-08-26T13:50:09.938 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:2400 IOs, 1104291840 bytes 2023-08-26T13:50:10.079 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:2500 IOs, 1127059456 bytes 2023-08-26T13:50:10.231 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:2600 IOs, 1144827904 bytes 2023-08-26T13:50:10.446 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:2700 IOs, 1202079744 bytes 2023-08-26T13:50:10.684 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:2800 IOs, 1293633536 bytes 2023-08-26T13:50:10.876 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:2900 IOs, 1314185216 bytes 2023-08-26T13:50:10.994 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:3000 IOs, 1322426368 bytes 2023-08-26T13:50:11.104 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:3100 IOs, 1323020288 bytes 2023-08-26T13:50:11.270 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:3200 IOs, 1368480768 bytes 2023-08-26T13:50:11.422 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:3300 IOs, 1382176768 bytes 2023-08-26T13:50:11.686 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:3400 IOs, 1454963712 bytes 2023-08-26T13:50:11.825 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:3500 IOs, 1469409280 bytes 2023-08-26T13:50:11.953 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:3600 IOs, 1484889088 bytes 2023-08-26T13:50:12.135 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:3700 IOs, 1522379776 bytes 2023-08-26T13:50:12.270 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:3800 IOs, 1538239488 bytes 2023-08-26T13:50:12.472 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:3900 IOs, 1581455360 bytes 2023-08-26T13:50:12.620 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:4000 IOs, 1601475584 bytes 2023-08-26T13:50:12.775 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:4100 IOs, 1624569856 bytes 2023-08-26T13:50:13.084 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:4200 IOs, 1728397312 bytes 2023-08-26T13:50:13.277 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:4300 IOs, 1774139392 bytes 2023-08-26T13:50:13.473 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:4400 IOs, 1816419328 bytes 2023-08-26T13:50:13.603 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:4500 IOs, 1835346944 bytes 2023-08-26T13:50:13.703 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:4600 IOs, 1842290688 bytes 2023-08-26T13:50:13.820 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:4700 IOs, 1842905088 bytes 2023-08-26T13:50:13.930 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:4800 IOs, 1843509248 bytes 2023-08-26T13:50:14.117 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:4900 IOs, 1888239616 bytes 2023-08-26T13:50:14.247 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:5000 IOs, 1896776704 bytes 2023-08-26T13:50:14.389 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:5100 IOs, 1915409408 bytes 2023-08-26T13:50:14.606 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:5200 IOs, 1969570816 bytes 2023-08-26T13:50:14.884 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:5300 IOs, 2044383232 bytes 2023-08-26T13:50:15.122 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:5400 IOs, 2112688128 bytes 2023-08-26T13:50:15.145 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ error_code=124 2023-08-26T13:50:15.145 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ set -e 2023-08-26T13:50:15.145 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ '[' 124 -eq 124 ']' 2023-08-26T13:50:15.145 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ return 0 2023-08-26T13:50:15.146 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-08-26T13:50:15.146 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ image=image_45 2023-08-26T13:50:15.146 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ create_image_and_enable_mirror cluster2 mirror image_45 snapshot 128M 2023-08-26T13:50:15.146 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster2 2023-08-26T13:50:15.146 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ shift 2023-08-26T13:50:15.147 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T13:50:15.147 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ shift 2023-08-26T13:50:15.147 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=image_45 2023-08-26T13:50:15.147 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ shift 2023-08-26T13:50:15.147 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local mode=snapshot 2023-08-26T13:50:15.148 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ '[' -n snapshot ']' 2023-08-26T13:50:15.148 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ shift 2023-08-26T13:50:15.148 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ create_image cluster2 mirror image_45 128M 2023-08-26T13:50:15.148 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster2 2023-08-26T13:50:15.148 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ shift 2023-08-26T13:50:15.149 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T13:50:15.149 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ shift 2023-08-26T13:50:15.149 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=image_45 2023-08-26T13:50:15.149 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ shift 2023-08-26T13:50:15.149 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local size=128 2023-08-26T13:50:15.149 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ '[' -n 128M ']' 2023-08-26T13:50:15.150 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ size=128M 2023-08-26T13:50:15.150 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ shift 2023-08-26T13:50:15.150 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock mirror/image_45 2023-08-26T13:50:15.190 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ '[' image = image ']' 2023-08-26T13:50:15.190 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ enable_mirror cluster2 mirror image_45 snapshot 2023-08-26T13:50:15.191 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster2 2023-08-26T13:50:15.191 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T13:50:15.191 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=image_45 2023-08-26T13:50:15.191 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local mode=snapshot 2023-08-26T13:50:15.192 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_45 snapshot 2023-08-26T13:50:16.116 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:Mirroring enabled 2023-08-26T13:50:16.124 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rbd --cluster=cluster2 info mirror/image_45 2023-08-26T13:50:16.163 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:rbd image 'image_45': 2023-08-26T13:50:16.163 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: size 128 MiB in 32 objects 2023-08-26T13:50:16.163 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: order 22 (4 MiB objects) 2023-08-26T13:50:16.164 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: snapshot_count: 1 2023-08-26T13:50:16.164 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: id: 142f47a901af 2023-08-26T13:50:16.164 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: block_name_prefix: rbd_data.142f47a901af 2023-08-26T13:50:16.164 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: format: 2 2023-08-26T13:50:16.164 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: features: layering, exclusive-lock 2023-08-26T13:50:16.165 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: op_features: 2023-08-26T13:50:16.165 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: flags: 2023-08-26T13:50:16.165 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: create_timestamp: Sat Aug 26 13:50:15 2023 2023-08-26T13:50:16.165 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: access_timestamp: Sat Aug 26 13:50:15 2023 2023-08-26T13:50:16.165 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: modify_timestamp: Sat Aug 26 13:50:15 2023 2023-08-26T13:50:16.166 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: mirroring state: enabled 2023-08-26T13:50:16.166 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: mirroring mode: snapshot 2023-08-26T13:50:16.166 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: mirroring global id: 1822564f-3eee-4ba8-b76e-97584ef869b2 2023-08-26T13:50:16.166 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: mirroring primary: true 2023-08-26T13:50:16.167 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ '[' -n '' ']' 2023-08-26T13:50:16.167 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ stress_write_image cluster2 mirror image_45 2023-08-26T13:50:16.167 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster2 2023-08-26T13:50:16.168 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T13:50:16.168 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=image_45 2023-08-26T13:50:16.168 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2023-08-26T13:50:16.169 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local duration=6 2023-08-26T13:50:16.169 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ set +e 2023-08-26T13:50:16.169 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ timeout 6s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_45 --debug-rbd=20 --debug-journaler=20 2023-08-26T13:50:16.340 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:100 IOs, 8541184 bytes 2023-08-26T13:50:16.491 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:200 IOs, 36551680 bytes 2023-08-26T13:50:16.680 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:300 IOs, 92342272 bytes 2023-08-26T13:50:16.892 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:400 IOs, 141384704 bytes 2023-08-26T13:50:17.052 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:500 IOs, 172513280 bytes 2023-08-26T13:50:17.269 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:600 IOs, 222137344 bytes 2023-08-26T13:50:17.507 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:700 IOs, 294928384 bytes 2023-08-26T13:50:17.614 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:800 IOs, 295563264 bytes 2023-08-26T13:50:17.737 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:900 IOs, 304175104 bytes 2023-08-26T13:50:17.929 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:1000 IOs, 350242816 bytes 2023-08-26T13:50:18.090 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:1100 IOs, 387517440 bytes 2023-08-26T13:50:18.342 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:1200 IOs, 447758336 bytes 2023-08-26T13:50:18.477 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:1300 IOs, 459660288 bytes 2023-08-26T13:50:18.664 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:1400 IOs, 507009024 bytes 2023-08-26T13:50:18.859 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:1500 IOs, 548562944 bytes 2023-08-26T13:50:19.011 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:1600 IOs, 577291264 bytes 2023-08-26T13:50:19.335 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:1700 IOs, 683954176 bytes 2023-08-26T13:50:19.458 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:1800 IOs, 692264960 bytes 2023-08-26T13:50:19.704 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:1900 IOs, 768584704 bytes 2023-08-26T13:50:19.934 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:2000 IOs, 832243712 bytes 2023-08-26T13:50:20.238 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:2100 IOs, 926944256 bytes 2023-08-26T13:50:20.395 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:2200 IOs, 951123968 bytes 2023-08-26T13:50:20.540 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:2300 IOs, 973007872 bytes 2023-08-26T13:50:20.793 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:2400 IOs, 1049696256 bytes 2023-08-26T13:50:21.039 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:2500 IOs, 1120835584 bytes 2023-08-26T13:50:21.255 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:2600 IOs, 1170345984 bytes 2023-08-26T13:50:21.373 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:2700 IOs, 1175481344 bytes 2023-08-26T13:50:21.540 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:2800 IOs, 1201329152 bytes 2023-08-26T13:50:21.671 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:2900 IOs, 1214882816 bytes 2023-08-26T13:50:21.782 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:3000 IOs, 1218713600 bytes 2023-08-26T13:50:21.941 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:3100 IOs, 1234337792 bytes 2023-08-26T13:50:22.173 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ error_code=124 2023-08-26T13:50:22.173 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ set -e 2023-08-26T13:50:22.174 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ '[' 124 -eq 124 ']' 2023-08-26T13:50:22.174 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ return 0 2023-08-26T13:50:22.174 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-08-26T13:50:22.174 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ image=image_46 2023-08-26T13:50:22.174 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ create_image_and_enable_mirror cluster2 mirror image_46 snapshot 128M 2023-08-26T13:50:22.175 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster2 2023-08-26T13:50:22.175 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ shift 2023-08-26T13:50:22.175 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T13:50:22.175 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ shift 2023-08-26T13:50:22.175 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=image_46 2023-08-26T13:50:22.175 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ shift 2023-08-26T13:50:22.176 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local mode=snapshot 2023-08-26T13:50:22.176 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ '[' -n snapshot ']' 2023-08-26T13:50:22.176 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ shift 2023-08-26T13:50:22.176 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ create_image cluster2 mirror image_46 128M 2023-08-26T13:50:22.176 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster2 2023-08-26T13:50:22.177 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ shift 2023-08-26T13:50:22.177 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T13:50:22.177 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ shift 2023-08-26T13:50:22.177 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=image_46 2023-08-26T13:50:22.177 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ shift 2023-08-26T13:50:22.178 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local size=128 2023-08-26T13:50:22.178 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ '[' -n 128M ']' 2023-08-26T13:50:22.178 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ size=128M 2023-08-26T13:50:22.178 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ shift 2023-08-26T13:50:22.178 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock mirror/image_46 2023-08-26T13:50:22.220 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ '[' image = image ']' 2023-08-26T13:50:22.221 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ enable_mirror cluster2 mirror image_46 snapshot 2023-08-26T13:50:22.221 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster2 2023-08-26T13:50:22.221 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T13:50:22.221 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=image_46 2023-08-26T13:50:22.221 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local mode=snapshot 2023-08-26T13:50:22.222 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_46 snapshot 2023-08-26T13:50:23.163 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:Mirroring enabled 2023-08-26T13:50:23.171 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rbd --cluster=cluster2 info mirror/image_46 2023-08-26T13:50:23.209 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:rbd image 'image_46': 2023-08-26T13:50:23.210 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: size 128 MiB in 32 objects 2023-08-26T13:50:23.210 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: order 22 (4 MiB objects) 2023-08-26T13:50:23.210 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: snapshot_count: 1 2023-08-26T13:50:23.210 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: id: 14376309b603 2023-08-26T13:50:23.210 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: block_name_prefix: rbd_data.14376309b603 2023-08-26T13:50:23.211 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: format: 2 2023-08-26T13:50:23.211 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: features: layering, exclusive-lock 2023-08-26T13:50:23.211 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: op_features: 2023-08-26T13:50:23.211 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: flags: 2023-08-26T13:50:23.212 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: create_timestamp: Sat Aug 26 13:50:22 2023 2023-08-26T13:50:23.212 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: access_timestamp: Sat Aug 26 13:50:22 2023 2023-08-26T13:50:23.212 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: modify_timestamp: Sat Aug 26 13:50:22 2023 2023-08-26T13:50:23.212 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: mirroring state: enabled 2023-08-26T13:50:23.212 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: mirroring mode: snapshot 2023-08-26T13:50:23.212 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: mirroring global id: 994498aa-fae7-419f-8291-f66b876aef55 2023-08-26T13:50:23.213 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: mirroring primary: true 2023-08-26T13:50:23.213 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ '[' -n '' ']' 2023-08-26T13:50:23.214 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ stress_write_image cluster2 mirror image_46 2023-08-26T13:50:23.214 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster2 2023-08-26T13:50:23.214 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T13:50:23.214 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=image_46 2023-08-26T13:50:23.214 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2023-08-26T13:50:23.215 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local duration=10 2023-08-26T13:50:23.215 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ set +e 2023-08-26T13:50:23.215 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ timeout 10s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_46 --debug-rbd=20 --debug-journaler=20 2023-08-26T13:50:23.453 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:100 IOs, 44508160 bytes 2023-08-26T13:50:23.759 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:200 IOs, 69321728 bytes 2023-08-26T13:50:23.940 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:300 IOs, 108390400 bytes 2023-08-26T13:50:24.074 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:400 IOs, 128823296 bytes 2023-08-26T13:50:24.329 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:500 IOs, 172224512 bytes 2023-08-26T13:50:24.533 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:600 IOs, 216700928 bytes 2023-08-26T13:50:24.738 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:700 IOs, 281327616 bytes 2023-08-26T13:50:24.955 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:800 IOs, 326942720 bytes 2023-08-26T13:50:25.204 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:900 IOs, 398841856 bytes 2023-08-26T13:50:25.364 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:1000 IOs, 425007104 bytes 2023-08-26T13:50:25.603 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:1100 IOs, 493641728 bytes 2023-08-26T13:50:25.852 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:1200 IOs, 568691712 bytes 2023-08-26T13:50:26.009 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:1300 IOs, 606612480 bytes 2023-08-26T13:50:26.227 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:1400 IOs, 631479296 bytes 2023-08-26T13:50:26.378 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:1500 IOs, 657269760 bytes 2023-08-26T13:50:26.606 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:1600 IOs, 713420800 bytes 2023-08-26T13:50:26.762 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:1700 IOs, 738699264 bytes 2023-08-26T13:50:26.948 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:1800 IOs, 780968960 bytes 2023-08-26T13:50:27.150 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:1900 IOs, 834320384 bytes 2023-08-26T13:50:27.354 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:2000 IOs, 873744384 bytes 2023-08-26T13:50:27.612 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:2100 IOs, 950806528 bytes 2023-08-26T13:50:27.838 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:2200 IOs, 1014146048 bytes 2023-08-26T13:50:28.032 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:2300 IOs, 1065656320 bytes 2023-08-26T13:50:28.209 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:2400 IOs, 1095596032 bytes 2023-08-26T13:50:28.397 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:2500 IOs, 1144467456 bytes 2023-08-26T13:50:28.524 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:2600 IOs, 1154221056 bytes 2023-08-26T13:50:28.712 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:2700 IOs, 1194759168 bytes 2023-08-26T13:50:28.822 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:2800 IOs, 1195367424 bytes 2023-08-26T13:50:28.968 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:2900 IOs, 1216908288 bytes 2023-08-26T13:50:29.305 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:3000 IOs, 1317760000 bytes 2023-08-26T13:50:29.484 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:3100 IOs, 1355795456 bytes 2023-08-26T13:50:29.643 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:3200 IOs, 1408831488 bytes 2023-08-26T13:50:29.905 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:3300 IOs, 1462850560 bytes 2023-08-26T13:50:30.012 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:3400 IOs, 1463457792 bytes 2023-08-26T13:50:30.148 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:3500 IOs, 1475259392 bytes 2023-08-26T13:50:30.285 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:3600 IOs, 1491002368 bytes 2023-08-26T13:50:30.532 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:3700 IOs, 1561181184 bytes 2023-08-26T13:50:30.690 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:3800 IOs, 1585717248 bytes 2023-08-26T13:50:30.922 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:3900 IOs, 1638275072 bytes 2023-08-26T13:50:31.209 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:4000 IOs, 1731435520 bytes 2023-08-26T13:50:31.406 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:4100 IOs, 1775975424 bytes 2023-08-26T13:50:31.613 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:4200 IOs, 1827179520 bytes 2023-08-26T13:50:31.746 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:4300 IOs, 1846420480 bytes 2023-08-26T13:50:31.927 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:4400 IOs, 1883993088 bytes 2023-08-26T13:50:32.136 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:4500 IOs, 1940662272 bytes 2023-08-26T13:50:32.315 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:4600 IOs, 1971255296 bytes 2023-08-26T13:50:32.552 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:4700 IOs, 2038120448 bytes 2023-08-26T13:50:32.789 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:4800 IOs, 2106548224 bytes 2023-08-26T13:50:32.964 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:4900 IOs, 2142237696 bytes 2023-08-26T13:50:33.144 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:5000 IOs, 2180675584 bytes 2023-08-26T13:50:33.221 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ error_code=124 2023-08-26T13:50:33.222 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ set -e 2023-08-26T13:50:33.222 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ '[' 124 -eq 124 ']' 2023-08-26T13:50:33.222 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ return 0 2023-08-26T13:50:33.222 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-08-26T13:50:33.222 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ image=image_47 2023-08-26T13:50:33.223 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ create_image_and_enable_mirror cluster2 mirror image_47 snapshot 128M 2023-08-26T13:50:33.223 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster2 2023-08-26T13:50:33.223 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ shift 2023-08-26T13:50:33.223 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T13:50:33.223 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ shift 2023-08-26T13:50:33.224 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=image_47 2023-08-26T13:50:33.224 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ shift 2023-08-26T13:50:33.224 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local mode=snapshot 2023-08-26T13:50:33.224 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ '[' -n snapshot ']' 2023-08-26T13:50:33.224 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ shift 2023-08-26T13:50:33.224 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ create_image cluster2 mirror image_47 128M 2023-08-26T13:50:33.225 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster2 2023-08-26T13:50:33.225 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ shift 2023-08-26T13:50:33.225 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T13:50:33.225 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ shift 2023-08-26T13:50:33.225 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=image_47 2023-08-26T13:50:33.226 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ shift 2023-08-26T13:50:33.226 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local size=128 2023-08-26T13:50:33.226 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ '[' -n 128M ']' 2023-08-26T13:50:33.226 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ size=128M 2023-08-26T13:50:33.226 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ shift 2023-08-26T13:50:33.227 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock mirror/image_47 2023-08-26T13:50:33.266 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ '[' image = image ']' 2023-08-26T13:50:33.267 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ enable_mirror cluster2 mirror image_47 snapshot 2023-08-26T13:50:33.267 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster2 2023-08-26T13:50:33.267 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T13:50:33.267 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=image_47 2023-08-26T13:50:33.268 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local mode=snapshot 2023-08-26T13:50:33.268 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_47 snapshot 2023-08-26T13:50:34.252 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:Mirroring enabled 2023-08-26T13:50:34.261 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rbd --cluster=cluster2 info mirror/image_47 2023-08-26T13:50:34.300 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:rbd image 'image_47': 2023-08-26T13:50:34.300 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: size 128 MiB in 32 objects 2023-08-26T13:50:34.300 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: order 22 (4 MiB objects) 2023-08-26T13:50:34.300 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: snapshot_count: 1 2023-08-26T13:50:34.301 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: id: 14413b7db149 2023-08-26T13:50:34.301 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: block_name_prefix: rbd_data.14413b7db149 2023-08-26T13:50:34.301 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: format: 2 2023-08-26T13:50:34.301 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: features: layering, exclusive-lock 2023-08-26T13:50:34.301 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: op_features: 2023-08-26T13:50:34.302 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: flags: 2023-08-26T13:50:34.302 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: create_timestamp: Sat Aug 26 13:50:33 2023 2023-08-26T13:50:34.302 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: access_timestamp: Sat Aug 26 13:50:33 2023 2023-08-26T13:50:34.302 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: modify_timestamp: Sat Aug 26 13:50:33 2023 2023-08-26T13:50:34.302 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: mirroring state: enabled 2023-08-26T13:50:34.303 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: mirroring mode: snapshot 2023-08-26T13:50:34.303 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: mirroring global id: 530e7abb-9a3e-4245-a312-8ee5a47a865b 2023-08-26T13:50:34.303 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: mirroring primary: true 2023-08-26T13:50:34.304 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ '[' -n '' ']' 2023-08-26T13:50:34.304 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ stress_write_image cluster2 mirror image_47 2023-08-26T13:50:34.304 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster2 2023-08-26T13:50:34.305 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T13:50:34.305 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=image_47 2023-08-26T13:50:34.305 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2023-08-26T13:50:34.306 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local duration=11 2023-08-26T13:50:34.306 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ set +e 2023-08-26T13:50:34.307 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ timeout 11s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_47 --debug-rbd=20 --debug-journaler=20 2023-08-26T13:50:34.552 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:100 IOs, 48676864 bytes 2023-08-26T13:50:34.752 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:200 IOs, 112261120 bytes 2023-08-26T13:50:34.971 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:300 IOs, 166666240 bytes 2023-08-26T13:50:35.134 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:400 IOs, 175839232 bytes 2023-08-26T13:50:35.412 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:500 IOs, 261787648 bytes 2023-08-26T13:50:35.616 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:600 IOs, 314255360 bytes 2023-08-26T13:50:35.810 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:700 IOs, 370210816 bytes 2023-08-26T13:50:36.044 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:800 IOs, 420990976 bytes 2023-08-26T13:50:36.262 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:900 IOs, 475742208 bytes 2023-08-26T13:50:36.410 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:1000 IOs, 496160768 bytes 2023-08-26T13:50:36.709 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:1100 IOs, 596403200 bytes 2023-08-26T13:50:36.930 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:1200 IOs, 646481920 bytes 2023-08-26T13:50:37.179 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:1300 IOs, 718237696 bytes 2023-08-26T13:50:37.386 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:1400 IOs, 773238784 bytes 2023-08-26T13:50:37.606 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:1500 IOs, 829465600 bytes 2023-08-26T13:50:37.748 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:1600 IOs, 845737984 bytes 2023-08-26T13:50:37.887 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:1700 IOs, 866288640 bytes 2023-08-26T13:50:38.133 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:1800 IOs, 934642688 bytes 2023-08-26T13:50:38.268 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:1900 IOs, 949278720 bytes 2023-08-26T13:50:38.380 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:2000 IOs, 949881856 bytes 2023-08-26T13:50:38.592 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:2100 IOs, 1005050880 bytes 2023-08-26T13:50:38.837 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:2200 IOs, 1073032192 bytes 2023-08-26T13:50:39.115 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:2300 IOs, 1167982592 bytes 2023-08-26T13:50:39.269 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:2400 IOs, 1169179648 bytes 2023-08-26T13:50:39.446 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:2500 IOs, 1195471872 bytes 2023-08-26T13:50:39.665 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:2600 IOs, 1242862592 bytes 2023-08-26T13:50:39.796 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:2700 IOs, 1259379712 bytes 2023-08-26T13:50:40.026 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:2800 IOs, 1320846336 bytes 2023-08-26T13:50:40.196 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:2900 IOs, 1354045440 bytes 2023-08-26T13:50:40.469 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:3000 IOs, 1434521600 bytes 2023-08-26T13:50:40.707 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:3100 IOs, 1508027392 bytes 2023-08-26T13:50:41.010 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:3200 IOs, 1588845568 bytes 2023-08-26T13:50:41.164 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:3300 IOs, 1609948160 bytes 2023-08-26T13:50:41.300 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:3400 IOs, 1627405312 bytes 2023-08-26T13:50:41.512 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:3500 IOs, 1688196096 bytes 2023-08-26T13:50:41.658 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:3600 IOs, 1704303616 bytes 2023-08-26T13:50:42.160 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:3700 IOs, 1909454848 bytes 2023-08-26T13:50:42.359 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:3800 IOs, 1961040896 bytes 2023-08-26T13:50:42.465 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:3900 IOs, 1961654272 bytes 2023-08-26T13:50:42.597 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:4000 IOs, 1972853760 bytes 2023-08-26T13:50:42.757 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:4100 IOs, 1995624448 bytes 2023-08-26T13:50:42.917 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:4200 IOs, 2021058560 bytes 2023-08-26T13:50:43.119 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:4300 IOs, 2072792064 bytes 2023-08-26T13:50:43.309 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:4400 IOs, 2117214208 bytes 2023-08-26T13:50:43.425 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:4500 IOs, 2123265024 bytes 2023-08-26T13:50:43.609 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:4600 IOs, 2161652736 bytes 2023-08-26T13:50:43.870 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:4700 IOs, 2240838656 bytes 2023-08-26T13:50:43.982 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:4800 IOs, 2241457152 bytes 2023-08-26T13:50:44.130 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:4900 IOs, 2262495232 bytes 2023-08-26T13:50:44.348 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:5000 IOs, 2320765952 bytes 2023-08-26T13:50:44.475 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:5100 IOs, 2333422592 bytes 2023-08-26T13:50:44.650 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:5200 IOs, 2369634304 bytes 2023-08-26T13:50:44.799 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:5300 IOs, 2397084672 bytes 2023-08-26T13:50:44.986 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:5400 IOs, 2441926656 bytes 2023-08-26T13:50:45.147 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:5500 IOs, 2468345856 bytes 2023-08-26T13:50:45.311 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ error_code=124 2023-08-26T13:50:45.311 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ set -e 2023-08-26T13:50:45.312 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ '[' 124 -eq 124 ']' 2023-08-26T13:50:45.312 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ return 0 2023-08-26T13:50:45.312 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-08-26T13:50:45.313 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ image=image_48 2023-08-26T13:50:45.313 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ create_image_and_enable_mirror cluster2 mirror image_48 snapshot 128M 2023-08-26T13:50:45.313 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster2 2023-08-26T13:50:45.313 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ shift 2023-08-26T13:50:45.313 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T13:50:45.314 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ shift 2023-08-26T13:50:45.314 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=image_48 2023-08-26T13:50:45.314 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ shift 2023-08-26T13:50:45.314 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local mode=snapshot 2023-08-26T13:50:45.314 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ '[' -n snapshot ']' 2023-08-26T13:50:45.315 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ shift 2023-08-26T13:50:45.315 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ create_image cluster2 mirror image_48 128M 2023-08-26T13:50:45.315 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster2 2023-08-26T13:50:45.315 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ shift 2023-08-26T13:50:45.315 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T13:50:45.316 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ shift 2023-08-26T13:50:45.316 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=image_48 2023-08-26T13:50:45.316 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ shift 2023-08-26T13:50:45.316 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local size=128 2023-08-26T13:50:45.316 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ '[' -n 128M ']' 2023-08-26T13:50:45.317 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ size=128M 2023-08-26T13:50:45.317 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ shift 2023-08-26T13:50:45.317 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock mirror/image_48 2023-08-26T13:50:45.356 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ '[' image = image ']' 2023-08-26T13:50:45.356 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ enable_mirror cluster2 mirror image_48 snapshot 2023-08-26T13:50:45.357 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster2 2023-08-26T13:50:45.357 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T13:50:45.357 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=image_48 2023-08-26T13:50:45.357 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local mode=snapshot 2023-08-26T13:50:45.357 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_48 snapshot 2023-08-26T13:50:46.302 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:Mirroring enabled 2023-08-26T13:50:46.310 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rbd --cluster=cluster2 info mirror/image_48 2023-08-26T13:50:46.346 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:rbd image 'image_48': 2023-08-26T13:50:46.346 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: size 128 MiB in 32 objects 2023-08-26T13:50:46.346 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: order 22 (4 MiB objects) 2023-08-26T13:50:46.347 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: snapshot_count: 1 2023-08-26T13:50:46.347 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: id: 144a5538e815 2023-08-26T13:50:46.347 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: block_name_prefix: rbd_data.144a5538e815 2023-08-26T13:50:46.347 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: format: 2 2023-08-26T13:50:46.348 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: features: layering, exclusive-lock 2023-08-26T13:50:46.348 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: op_features: 2023-08-26T13:50:46.348 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: flags: 2023-08-26T13:50:46.348 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: create_timestamp: Sat Aug 26 13:50:45 2023 2023-08-26T13:50:46.349 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: access_timestamp: Sat Aug 26 13:50:45 2023 2023-08-26T13:50:46.349 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: modify_timestamp: Sat Aug 26 13:50:45 2023 2023-08-26T13:50:46.349 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: mirroring state: enabled 2023-08-26T13:50:46.349 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: mirroring mode: snapshot 2023-08-26T13:50:46.349 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: mirroring global id: 6721bd5f-42ff-4eab-84c6-4ca33fff75c0 2023-08-26T13:50:46.350 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: mirroring primary: true 2023-08-26T13:50:46.351 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ '[' -n '' ']' 2023-08-26T13:50:46.351 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ stress_write_image cluster2 mirror image_48 2023-08-26T13:50:46.351 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster2 2023-08-26T13:50:46.351 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T13:50:46.352 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=image_48 2023-08-26T13:50:46.352 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2023-08-26T13:50:46.352 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local duration=8 2023-08-26T13:50:46.352 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ set +e 2023-08-26T13:50:46.352 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ timeout 8s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_48 --debug-rbd=20 --debug-journaler=20 2023-08-26T13:50:46.637 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:100 IOs, 67570688 bytes 2023-08-26T13:50:46.832 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:200 IOs, 116015104 bytes 2023-08-26T13:50:47.033 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:300 IOs, 187876352 bytes 2023-08-26T13:50:47.310 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:400 IOs, 261571584 bytes 2023-08-26T13:50:47.581 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:500 IOs, 355490816 bytes 2023-08-26T13:50:47.746 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:600 IOs, 379449344 bytes 2023-08-26T13:50:47.914 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:700 IOs, 409193472 bytes 2023-08-26T13:50:48.079 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:800 IOs, 434553856 bytes 2023-08-26T13:50:48.381 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:900 IOs, 539663360 bytes 2023-08-26T13:50:48.650 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:1000 IOs, 613160960 bytes 2023-08-26T13:50:48.863 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:1100 IOs, 666027008 bytes 2023-08-26T13:50:49.038 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:1200 IOs, 706054144 bytes 2023-08-26T13:50:49.199 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:1300 IOs, 727564288 bytes 2023-08-26T13:50:49.374 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:1400 IOs, 754947072 bytes 2023-08-26T13:50:49.610 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:1500 IOs, 819903488 bytes 2023-08-26T13:50:49.740 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:1600 IOs, 826582016 bytes 2023-08-26T13:50:49.985 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:1700 IOs, 891522048 bytes 2023-08-26T13:50:50.151 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:1800 IOs, 927619072 bytes 2023-08-26T13:50:50.374 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:1900 IOs, 985708544 bytes 2023-08-26T13:50:50.560 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:2000 IOs, 1018522624 bytes 2023-08-26T13:50:50.702 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:2100 IOs, 1030780928 bytes 2023-08-26T13:50:50.983 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:2200 IOs, 1119661056 bytes 2023-08-26T13:50:51.228 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:2300 IOs, 1207293952 bytes 2023-08-26T13:50:51.416 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:2400 IOs, 1249797120 bytes 2023-08-26T13:50:51.697 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:2500 IOs, 1325537280 bytes 2023-08-26T13:50:51.804 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:2600 IOs, 1326143488 bytes 2023-08-26T13:50:52.036 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:2700 IOs, 1389545472 bytes 2023-08-26T13:50:52.232 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:2800 IOs, 1422579712 bytes 2023-08-26T13:50:52.410 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:2900 IOs, 1460245504 bytes 2023-08-26T13:50:52.596 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:3000 IOs, 1517488128 bytes 2023-08-26T13:50:52.789 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:3100 IOs, 1537696768 bytes 2023-08-26T13:50:52.889 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:3200 IOs, 1538337792 bytes 2023-08-26T13:50:53.054 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:3300 IOs, 1563700224 bytes 2023-08-26T13:50:53.222 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:3400 IOs, 1594715136 bytes 2023-08-26T13:50:53.483 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:3500 IOs, 1631269888 bytes 2023-08-26T13:50:53.611 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:3600 IOs, 1640328192 bytes 2023-08-26T13:50:53.837 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:3700 IOs, 1707771904 bytes 2023-08-26T13:50:54.059 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:3800 IOs, 1748377600 bytes 2023-08-26T13:50:54.226 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:3900 IOs, 1779479552 bytes 2023-08-26T13:50:54.357 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ error_code=124 2023-08-26T13:50:54.358 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ set -e 2023-08-26T13:50:54.358 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ '[' 124 -eq 124 ']' 2023-08-26T13:50:54.358 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ return 0 2023-08-26T13:50:54.358 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-08-26T13:50:54.358 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ image=image_49 2023-08-26T13:50:54.359 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ create_image_and_enable_mirror cluster2 mirror image_49 snapshot 128M 2023-08-26T13:50:54.359 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster2 2023-08-26T13:50:54.359 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ shift 2023-08-26T13:50:54.359 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T13:50:54.359 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ shift 2023-08-26T13:50:54.360 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=image_49 2023-08-26T13:50:54.360 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ shift 2023-08-26T13:50:54.360 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local mode=snapshot 2023-08-26T13:50:54.360 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ '[' -n snapshot ']' 2023-08-26T13:50:54.360 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ shift 2023-08-26T13:50:54.361 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ create_image cluster2 mirror image_49 128M 2023-08-26T13:50:54.361 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster2 2023-08-26T13:50:54.361 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ shift 2023-08-26T13:50:54.361 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T13:50:54.361 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ shift 2023-08-26T13:50:54.361 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=image_49 2023-08-26T13:50:54.362 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ shift 2023-08-26T13:50:54.362 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local size=128 2023-08-26T13:50:54.362 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ '[' -n 128M ']' 2023-08-26T13:50:54.362 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ size=128M 2023-08-26T13:50:54.363 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ shift 2023-08-26T13:50:54.363 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock mirror/image_49 2023-08-26T13:50:54.405 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ '[' image = image ']' 2023-08-26T13:50:54.405 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ enable_mirror cluster2 mirror image_49 snapshot 2023-08-26T13:50:54.405 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster2 2023-08-26T13:50:54.405 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T13:50:54.405 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=image_49 2023-08-26T13:50:54.406 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local mode=snapshot 2023-08-26T13:50:54.406 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_49 snapshot 2023-08-26T13:50:55.394 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:Mirroring enabled 2023-08-26T13:50:55.404 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rbd --cluster=cluster2 info mirror/image_49 2023-08-26T13:50:55.644 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:rbd image 'image_49': 2023-08-26T13:50:55.644 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: size 128 MiB in 32 objects 2023-08-26T13:50:55.645 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: order 22 (4 MiB objects) 2023-08-26T13:50:55.645 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: snapshot_count: 1 2023-08-26T13:50:55.645 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: id: 1454473c9026 2023-08-26T13:50:55.645 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: block_name_prefix: rbd_data.1454473c9026 2023-08-26T13:50:55.645 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: format: 2 2023-08-26T13:50:55.646 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: features: layering, exclusive-lock 2023-08-26T13:50:55.646 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: op_features: 2023-08-26T13:50:55.646 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: flags: 2023-08-26T13:50:55.646 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: create_timestamp: Sat Aug 26 13:50:54 2023 2023-08-26T13:50:55.646 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: access_timestamp: Sat Aug 26 13:50:54 2023 2023-08-26T13:50:55.647 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: modify_timestamp: Sat Aug 26 13:50:54 2023 2023-08-26T13:50:55.647 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: mirroring state: enabled 2023-08-26T13:50:55.647 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: mirroring mode: snapshot 2023-08-26T13:50:55.647 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: mirroring global id: 5f1ee9ae-1ac1-4156-b437-c09951a7762e 2023-08-26T13:50:55.647 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: mirroring primary: true 2023-08-26T13:50:55.648 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ '[' -n '' ']' 2023-08-26T13:50:55.648 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ stress_write_image cluster2 mirror image_49 2023-08-26T13:50:55.649 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster2 2023-08-26T13:50:55.649 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T13:50:55.649 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=image_49 2023-08-26T13:50:55.649 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2023-08-26T13:50:55.650 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local duration=13 2023-08-26T13:50:55.650 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ set +e 2023-08-26T13:50:55.650 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ timeout 13s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_49 --debug-rbd=20 --debug-journaler=20 2023-08-26T13:50:56.086 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:100 IOs, 42911744 bytes 2023-08-26T13:50:56.249 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:200 IOs, 88828928 bytes 2023-08-26T13:50:56.449 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:300 IOs, 126797824 bytes 2023-08-26T13:50:56.578 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:400 IOs, 150381568 bytes 2023-08-26T13:50:56.723 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:500 IOs, 153273344 bytes 2023-08-26T13:50:56.870 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:600 IOs, 173416448 bytes 2023-08-26T13:50:57.088 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:700 IOs, 220319744 bytes 2023-08-26T13:50:57.212 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:800 IOs, 247008256 bytes 2023-08-26T13:50:57.428 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:900 IOs, 277203968 bytes 2023-08-26T13:50:57.543 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:1000 IOs, 303849472 bytes 2023-08-26T13:50:57.696 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:1100 IOs, 304434176 bytes 2023-08-26T13:50:57.869 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:1200 IOs, 336693248 bytes 2023-08-26T13:50:58.007 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:1300 IOs, 353314816 bytes 2023-08-26T13:50:58.229 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:1400 IOs, 404942848 bytes 2023-08-26T13:50:58.346 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:1500 IOs, 409539584 bytes 2023-08-26T13:50:58.522 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:1600 IOs, 439823360 bytes 2023-08-26T13:50:58.677 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:1700 IOs, 458781696 bytes 2023-08-26T13:50:58.905 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:1800 IOs, 518807552 bytes 2023-08-26T13:50:59.036 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:1900 IOs, 530735104 bytes 2023-08-26T13:50:59.143 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:2000 IOs, 531355648 bytes 2023-08-26T13:50:59.389 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:2100 IOs, 608942080 bytes 2023-08-26T13:50:59.511 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.0 with skip (0.5423683260161186) > weight (0.25) 2023-08-26T13:50:59.511 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.1 with skip (0.7146281688021794) > weight (0.25) 2023-08-26T13:50:59.512 INFO:tasks.rbd_mirror_thrash:kill cluster2.client.mirror.2 2023-08-26T13:50:59.512 INFO:tasks.rbd_mirror.cluster2.client.mirror.2:Sent signal 15 2023-08-26T13:50:59.512 INFO:tasks.rbd_mirror_thrash:waiting for 2 secs before reviving daemons 2023-08-26T13:50:59.513 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi124.stderr:2023-08-26T13:50:59.511+0000 7f186846c700 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.2 (PID: 117270) UID: 1000 2023-08-26T13:50:59.598 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:2200 IOs, 652667904 bytes 2023-08-26T13:50:59.709 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:2300 IOs, 653274112 bytes 2023-08-26T13:50:59.920 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:2400 IOs, 709651456 bytes 2023-08-26T13:51:00.097 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:2500 IOs, 739418112 bytes 2023-08-26T13:51:00.270 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:2600 IOs, 769224704 bytes 2023-08-26T13:51:00.392 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:2700 IOs, 776040448 bytes 2023-08-26T13:51:00.658 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:2800 IOs, 843165696 bytes 2023-08-26T13:51:01.023 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:2900 IOs, 892547072 bytes 2023-08-26T13:51:01.215 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:3000 IOs, 934323200 bytes 2023-08-26T13:51:01.366 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:3100 IOs, 955344896 bytes 2023-08-26T13:51:01.510 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:3200 IOs, 976770048 bytes 2023-08-26T13:51:01.513 INFO:tasks.rbd_mirror_thrash:waiting for cluster2.client.mirror.2 2023-08-26T13:51:01.513 INFO:teuthology.orchestra.run:waiting for 600 2023-08-26T13:51:01.514 INFO:tasks.rbd_mirror_thrash:reviving cluster2.client.mirror.2 2023-08-26T13:51:01.514 INFO:tasks.rbd_mirror.cluster2.client.mirror.2:Restarting daemon 2023-08-26T13:51:01.514 DEBUG:teuthology.orchestra.run.smithi124:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.2 2023-08-26T13:51:01.517 INFO:tasks.rbd_mirror.cluster2.client.mirror.2:Started 2023-08-26T13:51:01.517 INFO:tasks.rbd_mirror_thrash:waiting for 117 secs before thrashing 2023-08-26T13:51:01.674 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:3300 IOs, 1000650752 bytes 2023-08-26T13:51:01.792 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:3400 IOs, 1004692480 bytes 2023-08-26T13:51:02.001 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:3500 IOs, 1056996352 bytes 2023-08-26T13:51:02.276 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:3600 IOs, 1125474304 bytes 2023-08-26T13:51:02.457 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:3700 IOs, 1167583232 bytes 2023-08-26T13:51:02.633 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:3800 IOs, 1210312704 bytes 2023-08-26T13:51:02.816 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:3900 IOs, 1232451584 bytes 2023-08-26T13:51:03.031 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:4000 IOs, 1293007872 bytes 2023-08-26T13:51:03.143 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:4100 IOs, 1293618176 bytes 2023-08-26T13:51:03.247 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:4200 IOs, 1294234624 bytes 2023-08-26T13:51:03.397 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:4300 IOs, 1334672384 bytes 2023-08-26T13:51:03.618 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:4400 IOs, 1378385920 bytes 2023-08-26T13:51:03.788 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:4500 IOs, 1411717120 bytes 2023-08-26T13:51:04.056 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:4600 IOs, 1496131584 bytes 2023-08-26T13:51:04.301 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:4700 IOs, 1568180224 bytes 2023-08-26T13:51:04.480 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:4800 IOs, 1603659776 bytes 2023-08-26T13:51:04.721 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:4900 IOs, 1667931136 bytes 2023-08-26T13:51:04.724 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.0 with skip (0.3402959129851967) > weight (0.25) 2023-08-26T13:51:04.725 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.1 with skip (0.9398309651036275) > weight (0.25) 2023-08-26T13:51:04.725 INFO:tasks.rbd_mirror_thrash:kill cluster1.client.mirror.2 2023-08-26T13:51:04.726 INFO:tasks.rbd_mirror.cluster1.client.mirror.2:Sent signal 15 2023-08-26T13:51:04.726 INFO:tasks.rbd_mirror_thrash:waiting for 1 secs before reviving daemons 2023-08-26T13:51:04.727 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi124.stderr:2023-08-26T13:51:04.725+0000 7fea46b43700 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.2 (PID: 112526) UID: 1000 2023-08-26T13:51:04.922 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:5000 IOs, 1709380608 bytes 2023-08-26T13:51:05.107 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:5100 IOs, 1763589120 bytes 2023-08-26T13:51:05.322 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:5200 IOs, 1807228928 bytes 2023-08-26T13:51:05.465 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:5300 IOs, 1827036160 bytes 2023-08-26T13:51:05.583 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:5400 IOs, 1858777088 bytes 2023-08-26T13:51:05.728 INFO:tasks.rbd_mirror_thrash:waiting for cluster1.client.mirror.2 2023-08-26T13:51:05.728 INFO:teuthology.orchestra.run:waiting for 600 2023-08-26T13:51:05.728 INFO:tasks.rbd_mirror_thrash:reviving cluster1.client.mirror.2 2023-08-26T13:51:05.729 INFO:tasks.rbd_mirror.cluster1.client.mirror.2:Restarting daemon 2023-08-26T13:51:05.729 DEBUG:teuthology.orchestra.run.smithi124:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.2 2023-08-26T13:51:05.733 INFO:tasks.rbd_mirror.cluster1.client.mirror.2:Started 2023-08-26T13:51:05.733 INFO:tasks.rbd_mirror_thrash:waiting for 101 secs before thrashing 2023-08-26T13:51:05.761 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:5500 IOs, 1874051072 bytes 2023-08-26T13:51:05.938 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:5600 IOs, 1897757696 bytes 2023-08-26T13:51:06.114 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:5700 IOs, 1928780800 bytes 2023-08-26T13:51:06.246 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:5800 IOs, 1942500352 bytes 2023-08-26T13:51:06.434 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:5900 IOs, 1986153472 bytes 2023-08-26T13:51:06.580 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:6000 IOs, 2008197120 bytes 2023-08-26T13:51:06.794 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:6100 IOs, 2060065792 bytes 2023-08-26T13:51:06.976 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:6200 IOs, 2104842240 bytes 2023-08-26T13:51:07.114 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:6300 IOs, 2123691008 bytes 2023-08-26T13:51:07.288 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:6400 IOs, 2168285184 bytes 2023-08-26T13:51:07.359 INFO:tasks.ceph.cluster2.osd.0.smithi124.stderr:2023-08-26T13:51:07.356+0000 7fb60dfa7700 -1 reset not still connected to 0x564562f856c0 2023-08-26T13:51:07.475 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:6500 IOs, 2199458816 bytes 2023-08-26T13:51:07.637 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:6600 IOs, 2225183744 bytes 2023-08-26T13:51:07.863 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:6700 IOs, 2287188992 bytes 2023-08-26T13:51:07.981 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:6800 IOs, 2287793152 bytes 2023-08-26T13:51:08.221 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:6900 IOs, 2359575552 bytes 2023-08-26T13:51:08.534 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:7000 IOs, 2479711232 bytes 2023-08-26T13:51:08.656 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ error_code=124 2023-08-26T13:51:08.656 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ set -e 2023-08-26T13:51:08.657 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ '[' 124 -eq 124 ']' 2023-08-26T13:51:08.657 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ return 0 2023-08-26T13:51:08.657 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-08-26T13:51:08.657 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ image=image_50 2023-08-26T13:51:08.657 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ create_image_and_enable_mirror cluster2 mirror image_50 snapshot 128M 2023-08-26T13:51:08.658 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster2 2023-08-26T13:51:08.658 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ shift 2023-08-26T13:51:08.658 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T13:51:08.658 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ shift 2023-08-26T13:51:08.658 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=image_50 2023-08-26T13:51:08.659 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ shift 2023-08-26T13:51:08.659 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local mode=snapshot 2023-08-26T13:51:08.659 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ '[' -n snapshot ']' 2023-08-26T13:51:08.659 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ shift 2023-08-26T13:51:08.659 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ create_image cluster2 mirror image_50 128M 2023-08-26T13:51:08.660 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster2 2023-08-26T13:51:08.660 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ shift 2023-08-26T13:51:08.660 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T13:51:08.660 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ shift 2023-08-26T13:51:08.660 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=image_50 2023-08-26T13:51:08.660 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ shift 2023-08-26T13:51:08.661 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local size=128 2023-08-26T13:51:08.661 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ '[' -n 128M ']' 2023-08-26T13:51:08.661 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ size=128M 2023-08-26T13:51:08.661 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ shift 2023-08-26T13:51:08.661 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock mirror/image_50 2023-08-26T13:51:08.707 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ '[' image = image ']' 2023-08-26T13:51:08.707 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ enable_mirror cluster2 mirror image_50 snapshot 2023-08-26T13:51:08.707 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster2 2023-08-26T13:51:08.707 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T13:51:08.708 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=image_50 2023-08-26T13:51:08.708 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local mode=snapshot 2023-08-26T13:51:08.708 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_50 snapshot 2023-08-26T13:51:09.491 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:Mirroring enabled 2023-08-26T13:51:09.500 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rbd --cluster=cluster2 info mirror/image_50 2023-08-26T13:51:09.536 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:rbd image 'image_50': 2023-08-26T13:51:09.536 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: size 128 MiB in 32 objects 2023-08-26T13:51:09.537 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: order 22 (4 MiB objects) 2023-08-26T13:51:09.537 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: snapshot_count: 1 2023-08-26T13:51:09.537 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: id: 146cb9ccf0e9 2023-08-26T13:51:09.537 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: block_name_prefix: rbd_data.146cb9ccf0e9 2023-08-26T13:51:09.538 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: format: 2 2023-08-26T13:51:09.538 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: features: layering, exclusive-lock 2023-08-26T13:51:09.538 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: op_features: 2023-08-26T13:51:09.538 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: flags: 2023-08-26T13:51:09.538 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: create_timestamp: Sat Aug 26 13:51:08 2023 2023-08-26T13:51:09.539 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: access_timestamp: Sat Aug 26 13:51:08 2023 2023-08-26T13:51:09.539 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: modify_timestamp: Sat Aug 26 13:51:08 2023 2023-08-26T13:51:09.539 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: mirroring state: enabled 2023-08-26T13:51:09.539 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: mirroring mode: snapshot 2023-08-26T13:51:09.540 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: mirroring global id: ebb4d874-359b-4be6-9915-df44e455c46f 2023-08-26T13:51:09.540 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: mirroring primary: true 2023-08-26T13:51:09.542 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ '[' -n '' ']' 2023-08-26T13:51:09.542 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ stress_write_image cluster2 mirror image_50 2023-08-26T13:51:09.542 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster2 2023-08-26T13:51:09.543 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T13:51:09.543 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=image_50 2023-08-26T13:51:09.543 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2023-08-26T13:51:09.544 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local duration=11 2023-08-26T13:51:09.544 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ set +e 2023-08-26T13:51:09.544 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ timeout 11s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_50 --debug-rbd=20 --debug-journaler=20 2023-08-26T13:51:09.839 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:100 IOs, 83946496 bytes 2023-08-26T13:51:10.091 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:200 IOs, 156680192 bytes 2023-08-26T13:51:10.212 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:300 IOs, 170725376 bytes 2023-08-26T13:51:10.336 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:400 IOs, 198892544 bytes 2023-08-26T13:51:10.571 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:500 IOs, 250847232 bytes 2023-08-26T13:51:10.743 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:600 IOs, 283542528 bytes 2023-08-26T13:51:11.085 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:700 IOs, 401230848 bytes 2023-08-26T13:51:11.219 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:800 IOs, 406922240 bytes 2023-08-26T13:51:11.356 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:900 IOs, 419496960 bytes 2023-08-26T13:51:11.552 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:1000 IOs, 461510656 bytes 2023-08-26T13:51:11.703 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:1100 IOs, 481946624 bytes 2023-08-26T13:51:11.942 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:1200 IOs, 548144128 bytes 2023-08-26T13:51:12.188 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:1300 IOs, 605092864 bytes 2023-08-26T13:51:12.371 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:1400 IOs, 638539776 bytes 2023-08-26T13:51:12.521 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:1500 IOs, 653877248 bytes 2023-08-26T13:51:12.774 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:1600 IOs, 733765632 bytes 2023-08-26T13:51:12.962 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:1700 IOs, 774097920 bytes 2023-08-26T13:51:13.074 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:1800 IOs, 774723584 bytes 2023-08-26T13:51:13.269 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:1900 IOs, 820027392 bytes 2023-08-26T13:51:13.401 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:2000 IOs, 850366464 bytes 2023-08-26T13:51:13.655 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:2100 IOs, 909279232 bytes 2023-08-26T13:51:13.880 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:2200 IOs, 965352448 bytes 2023-08-26T13:51:14.005 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:2300 IOs, 981261312 bytes 2023-08-26T13:51:14.223 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:2400 IOs, 1040641024 bytes 2023-08-26T13:51:14.431 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:2500 IOs, 1087830016 bytes 2023-08-26T13:51:14.676 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:2600 IOs, 1151842304 bytes 2023-08-26T13:51:14.898 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:2700 IOs, 1207852032 bytes 2023-08-26T13:51:15.070 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:2800 IOs, 1239228416 bytes 2023-08-26T13:51:15.204 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:2900 IOs, 1255947264 bytes 2023-08-26T13:51:15.398 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:3000 IOs, 1296040960 bytes 2023-08-26T13:51:15.608 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:3100 IOs, 1343460352 bytes 2023-08-26T13:51:15.892 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:3200 IOs, 1434007552 bytes 2023-08-26T13:51:16.039 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:3300 IOs, 1476898816 bytes 2023-08-26T13:51:16.253 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:3400 IOs, 1503926272 bytes 2023-08-26T13:51:16.405 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:3500 IOs, 1522454528 bytes 2023-08-26T13:51:16.590 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:3600 IOs, 1570127872 bytes 2023-08-26T13:51:16.761 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:3700 IOs, 1591575552 bytes 2023-08-26T13:51:17.009 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:3800 IOs, 1675378688 bytes 2023-08-26T13:51:17.224 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:3900 IOs, 1725768704 bytes 2023-08-26T13:51:17.515 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:4000 IOs, 1819954176 bytes 2023-08-26T13:51:17.776 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:4100 IOs, 1896510464 bytes 2023-08-26T13:51:17.912 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:4200 IOs, 1911796736 bytes 2023-08-26T13:51:18.198 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:4300 IOs, 1986227200 bytes 2023-08-26T13:51:18.437 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:4400 IOs, 2077139968 bytes 2023-08-26T13:51:18.706 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:4500 IOs, 2137852928 bytes 2023-08-26T13:51:18.954 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:4600 IOs, 2209356800 bytes 2023-08-26T13:51:19.150 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:4700 IOs, 2253708288 bytes 2023-08-26T13:51:19.417 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:4800 IOs, 2332622848 bytes 2023-08-26T13:51:19.746 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:4900 IOs, 2358017024 bytes 2023-08-26T13:51:19.872 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:5000 IOs, 2379409408 bytes 2023-08-26T13:51:20.124 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:5100 IOs, 2444285952 bytes 2023-08-26T13:51:20.308 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:5200 IOs, 2485208064 bytes 2023-08-26T13:51:20.546 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ error_code=124 2023-08-26T13:51:20.546 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ set -e 2023-08-26T13:51:20.547 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ '[' 124 -eq 124 ']' 2023-08-26T13:51:20.547 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ return 0 2023-08-26T13:51:20.547 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ wait_for_pool_images cluster2 mirror 50 2023-08-26T13:51:20.547 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster2 2023-08-26T13:51:20.548 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T13:51:20.548 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image_count=50 2023-08-26T13:51:20.548 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local s 2023-08-26T13:51:20.548 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local count 2023-08-26T13:51:20.549 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local last_count=0 2023-08-26T13:51:20.549 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ true 2023-08-26T13:51:20.549 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ seq 1 40 2023-08-26T13:51:20.550 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ for s in `seq 1 40` 2023-08-26T13:51:20.550 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ test 1 -ne 1 2023-08-26T13:51:20.550 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ rbd --cluster cluster2 -p mirror mirror pool status 2023-08-26T13:51:20.551 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ grep 'images: ' 2023-08-26T13:51:20.551 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ cut '-d ' -f 2 2023-08-26T13:51:20.583 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:rbd: failed to query services: (13) Permission denied 2023-08-26T13:51:20.586 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ count=50 2023-08-26T13:51:20.586 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ test 50 = 50 2023-08-26T13:51:20.586 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ return 0 2023-08-26T13:51:20.586 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ wait_for_pool_healthy cluster2 mirror 2023-08-26T13:51:20.586 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster2 2023-08-26T13:51:20.587 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T13:51:20.587 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local s 2023-08-26T13:51:20.587 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local state 2023-08-26T13:51:20.587 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ seq 1 40 2023-08-26T13:51:20.588 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ for s in `seq 1 40` 2023-08-26T13:51:20.589 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ test 1 -ne 1 2023-08-26T13:51:20.589 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ rbd --cluster cluster2 -p mirror mirror pool status 2023-08-26T13:51:20.589 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ grep 'image health:' 2023-08-26T13:51:20.589 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ cut '-d ' -f 3 2023-08-26T13:51:20.621 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:rbd: failed to query services: (13) Permission denied 2023-08-26T13:51:20.624 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ state=WARNING 2023-08-26T13:51:20.624 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ test WARNING = ERROR 2023-08-26T13:51:20.625 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ test WARNING = OK 2023-08-26T13:51:20.625 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ for s in `seq 1 40` 2023-08-26T13:51:20.625 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ test 2 -ne 1 2023-08-26T13:51:20.625 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ sleep 30 2023-08-26T13:51:50.626 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ rbd --cluster cluster2 -p mirror mirror pool status 2023-08-26T13:51:50.626 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ grep 'image health:' 2023-08-26T13:51:50.626 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ cut '-d ' -f 3 2023-08-26T13:51:50.666 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:rbd: failed to query services: (13) Permission denied 2023-08-26T13:51:50.669 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ state=OK 2023-08-26T13:51:50.670 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ test OK = ERROR 2023-08-26T13:51:50.670 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ test OK = OK 2023-08-26T13:51:50.670 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ return 0 2023-08-26T13:51:50.670 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ wait_for_pool_images cluster1 mirror 50 2023-08-26T13:51:50.671 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster1 2023-08-26T13:51:50.671 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T13:51:50.671 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image_count=50 2023-08-26T13:51:50.671 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local s 2023-08-26T13:51:50.671 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local count 2023-08-26T13:51:50.672 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local last_count=0 2023-08-26T13:51:50.672 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ true 2023-08-26T13:51:50.672 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ seq 1 40 2023-08-26T13:51:50.673 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ for s in `seq 1 40` 2023-08-26T13:51:50.673 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ test 1 -ne 1 2023-08-26T13:51:50.673 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ rbd --cluster cluster1 -p mirror mirror pool status 2023-08-26T13:51:50.673 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ grep 'images: ' 2023-08-26T13:51:50.673 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ cut '-d ' -f 2 2023-08-26T13:51:50.710 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:rbd: failed to query services: (13) Permission denied 2023-08-26T13:51:50.713 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ count=50 2023-08-26T13:51:50.714 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ test 50 = 50 2023-08-26T13:51:50.714 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ return 0 2023-08-26T13:51:50.714 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ wait_for_pool_healthy cluster1 mirror 2023-08-26T13:51:50.714 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster1 2023-08-26T13:51:50.714 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T13:51:50.715 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local s 2023-08-26T13:51:50.715 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local state 2023-08-26T13:51:50.715 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ seq 1 40 2023-08-26T13:51:50.716 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ for s in `seq 1 40` 2023-08-26T13:51:50.716 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ test 1 -ne 1 2023-08-26T13:51:50.716 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ rbd --cluster cluster1 -p mirror mirror pool status 2023-08-26T13:51:50.716 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ grep 'image health:' 2023-08-26T13:51:50.717 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ cut '-d ' -f 3 2023-08-26T13:51:50.957 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:rbd: failed to query services: (13) Permission denied 2023-08-26T13:51:50.960 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ state=OK 2023-08-26T13:51:50.961 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ test OK = ERROR 2023-08-26T13:51:50.961 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ test OK = OK 2023-08-26T13:51:50.961 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ return 0 2023-08-26T13:51:50.961 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ testlog 'TEST: compare many images' 2023-08-26T13:51:50.962 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ tee -a /tmp/tmp.TfDn6BcQ3N/rbd-mirror.test.log 2023-08-26T13:51:50.962 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ date '+%F %T' 2023-08-26T13:51:50.963 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ echo 2023-08-26 13:51:50 TEST: compare many images 2023-08-26T13:51:50.963 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:2023-08-26 13:51:50 TEST: compare many images 2023-08-26T13:51:50.963 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ seq 1 50 2023-08-26T13:51:50.964 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-08-26T13:51:50.964 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ image=image_1 2023-08-26T13:51:50.964 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ create_snap cluster2 mirror image_1 snap 2023-08-26T13:51:50.964 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster2 2023-08-26T13:51:50.965 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T13:51:50.965 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=image_1 2023-08-26T13:51:50.965 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local snap_name=snap 2023-08-26T13:51:50.965 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rbd --cluster cluster2 -p mirror snap create image_1@snap --debug-rbd=20 --debug-journaler=20 2023-08-26T13:51:52.615 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ wait_for_image_replay_started cluster1 mirror image_1 2023-08-26T13:51:52.616 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster1 2023-08-26T13:51:52.616 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T13:51:52.616 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=image_1 2023-08-26T13:51:52.616 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ wait_for_image_replay_state cluster1 mirror image_1 started 2023-08-26T13:51:52.617 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster1 2023-08-26T13:51:52.617 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T13:51:52.617 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=image_1 2023-08-26T13:51:52.617 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local state=started 2023-08-26T13:51:52.617 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local s 2023-08-26T13:51:52.618 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-08-26T13:51:52.618 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ sleep 1 2023-08-26T13:51:53.616 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ test_image_replay_state cluster1 mirror image_1 started 2023-08-26T13:51:53.617 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster1 2023-08-26T13:51:53.617 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T13:51:53.617 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=image_1 2023-08-26T13:51:53.617 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local test_state=started 2023-08-26T13:51:53.617 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local status_result 2023-08-26T13:51:53.618 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local current_state=stopped 2023-08-26T13:51:53.619 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_1 2023-08-26T13:51:53.619 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local cluster_instance=cluster1 2023-08-26T13:51:53.619 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ shift 2023-08-26T13:51:53.619 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local cluster=cluster1 2023-08-26T13:51:53.619 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local instance=cluster1 2023-08-26T13:51:53.620 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local loop_instance 2023-08-26T13:51:53.620 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-08-26T13:51:53.620 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ sleep 0 2023-08-26T13:51:53.620 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ grep -i state 2023-08-26T13:51:53.620 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ '[' cluster1 '!=' cluster1 ']' 2023-08-26T13:51:53.621 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ seq 0 3 2023-08-26T13:51:53.622 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-08-26T13:51:53.623 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_1 2023-08-26T13:51:53.623 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local cluster=cluster1:0 2023-08-26T13:51:53.623 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ shift 2023-08-26T13:51:53.623 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local instance 2023-08-26T13:51:53.624 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ set_cluster_instance cluster1:0 cluster instance 2023-08-26T13:51:53.624 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local val=cluster1:0 2023-08-26T13:51:53.624 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local cluster_var_name=cluster 2023-08-26T13:51:53.624 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local instance_var_name=instance 2023-08-26T13:51:53.624 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ cluster=cluster1 2023-08-26T13:51:53.625 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ instance=0 2023-08-26T13:51:53.625 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ '[' 0 = cluster1:0 ']' 2023-08-26T13:51:53.625 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ eval cluster=cluster1 2023-08-26T13:51:53.625 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ cluster=cluster1 2023-08-26T13:51:53.625 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ eval instance=0 2023-08-26T13:51:53.625 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ instance=0 2023-08-26T13:51:53.626 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ daemon_asok_file cluster1:0 cluster1 2023-08-26T13:51:53.626 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ local local_cluster=cluster1:0 2023-08-26T13:51:53.626 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ local cluster=cluster1 2023-08-26T13:51:53.626 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ local instance 2023-08-26T13:51:53.626 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2023-08-26T13:51:53.627 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ local val=cluster1:0 2023-08-26T13:51:53.628 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ local cluster_var_name=local_cluster 2023-08-26T13:51:53.628 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ local instance_var_name=instance 2023-08-26T13:51:53.628 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ cluster=cluster1 2023-08-26T13:51:53.628 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ instance=0 2023-08-26T13:51:53.628 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ '[' 0 = cluster1:0 ']' 2023-08-26T13:51:53.629 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ eval local_cluster=cluster1 2023-08-26T13:51:53.629 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++++ local_cluster=cluster1 2023-08-26T13:51:53.629 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ eval instance=0 2023-08-26T13:51:53.629 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++++ instance=0 2023-08-26T13:51:53.629 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2023-08-26T13:51:53.644 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-08-26T13:51:53.644 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-08-26T13:51:53.644 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-08-26T13:51:53.644 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_1 2023-08-26T13:51:53.759 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:no valid command found; 3 closest matches: 2023-08-26T13:51:53.760 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:rbd mirror status mirror/image_10 2023-08-26T13:51:53.760 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:rbd mirror status mirror/image_12 2023-08-26T13:51:53.760 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:rbd mirror status mirror/image_16 2023-08-26T13:51:53.760 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:admin_socket: invalid command 2023-08-26T13:51:53.763 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-08-26T13:51:53.764 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_1 2023-08-26T13:51:53.764 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local cluster=cluster1:1 2023-08-26T13:51:53.764 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ shift 2023-08-26T13:51:53.764 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local instance 2023-08-26T13:51:53.764 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ set_cluster_instance cluster1:1 cluster instance 2023-08-26T13:51:53.765 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local val=cluster1:1 2023-08-26T13:51:53.765 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local cluster_var_name=cluster 2023-08-26T13:51:53.765 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local instance_var_name=instance 2023-08-26T13:51:53.765 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ cluster=cluster1 2023-08-26T13:51:53.765 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ instance=1 2023-08-26T13:51:53.765 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ '[' 1 = cluster1:1 ']' 2023-08-26T13:51:53.766 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ eval cluster=cluster1 2023-08-26T13:51:53.766 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ cluster=cluster1 2023-08-26T13:51:53.766 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ eval instance=1 2023-08-26T13:51:53.766 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ instance=1 2023-08-26T13:51:53.766 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ daemon_asok_file cluster1:1 cluster1 2023-08-26T13:51:53.767 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ local local_cluster=cluster1:1 2023-08-26T13:51:53.767 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ local cluster=cluster1 2023-08-26T13:51:53.767 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ local instance 2023-08-26T13:51:53.767 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2023-08-26T13:51:53.767 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ local val=cluster1:1 2023-08-26T13:51:53.767 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ local cluster_var_name=local_cluster 2023-08-26T13:51:53.768 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ local instance_var_name=instance 2023-08-26T13:51:53.768 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ cluster=cluster1 2023-08-26T13:51:53.768 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ instance=1 2023-08-26T13:51:53.768 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ '[' 1 = cluster1:1 ']' 2023-08-26T13:51:53.768 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ eval local_cluster=cluster1 2023-08-26T13:51:53.769 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++++ local_cluster=cluster1 2023-08-26T13:51:53.769 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ eval instance=1 2023-08-26T13:51:53.769 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++++ instance=1 2023-08-26T13:51:53.769 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2023-08-26T13:51:53.782 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-08-26T13:51:53.782 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-08-26T13:51:53.782 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-08-26T13:51:53.783 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_1 2023-08-26T13:51:53.888 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:no valid command found; 2 closest matches: 2023-08-26T13:51:53.888 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:rbd mirror status mirror/image_11 2023-08-26T13:51:53.889 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:rbd mirror status mirror/image_14 2023-08-26T13:51:53.889 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:admin_socket: invalid command 2023-08-26T13:51:53.891 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-08-26T13:51:53.891 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/image_1 2023-08-26T13:51:53.892 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local cluster=cluster1:2 2023-08-26T13:51:53.892 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ shift 2023-08-26T13:51:53.892 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local instance 2023-08-26T13:51:53.892 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ set_cluster_instance cluster1:2 cluster instance 2023-08-26T13:51:53.892 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local val=cluster1:2 2023-08-26T13:51:53.893 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local cluster_var_name=cluster 2023-08-26T13:51:53.893 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local instance_var_name=instance 2023-08-26T13:51:53.893 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ cluster=cluster1 2023-08-26T13:51:53.893 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ instance=2 2023-08-26T13:51:53.893 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ '[' 2 = cluster1:2 ']' 2023-08-26T13:51:53.894 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ eval cluster=cluster1 2023-08-26T13:51:53.894 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ cluster=cluster1 2023-08-26T13:51:53.894 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ eval instance=2 2023-08-26T13:51:53.894 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ instance=2 2023-08-26T13:51:53.894 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ daemon_asok_file cluster1:2 cluster1 2023-08-26T13:51:53.894 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ local local_cluster=cluster1:2 2023-08-26T13:51:53.895 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ local cluster=cluster1 2023-08-26T13:51:53.895 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ local instance 2023-08-26T13:51:53.895 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2023-08-26T13:51:53.895 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ local val=cluster1:2 2023-08-26T13:51:53.895 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ local cluster_var_name=local_cluster 2023-08-26T13:51:53.896 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ local instance_var_name=instance 2023-08-26T13:51:53.896 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ cluster=cluster1 2023-08-26T13:51:53.896 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ instance=2 2023-08-26T13:51:53.896 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ '[' 2 = cluster1:2 ']' 2023-08-26T13:51:53.896 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ eval local_cluster=cluster1 2023-08-26T13:51:53.897 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++++ local_cluster=cluster1 2023-08-26T13:51:53.897 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ eval instance=2 2023-08-26T13:51:53.897 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++++ instance=2 2023-08-26T13:51:53.897 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2023-08-26T13:51:53.911 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-08-26T13:51:53.911 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-08-26T13:51:53.911 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-08-26T13:51:53.911 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/image_1 2023-08-26T13:51:54.024 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ return 0 2023-08-26T13:51:54.024 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ status_result=' "state": "Replaying"' 2023-08-26T13:51:54.025 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ echo ' "state": "Replaying"' 2023-08-26T13:51:54.025 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ grep -i Replaying 2023-08-26T13:51:54.026 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: "state": "Replaying" 2023-08-26T13:51:54.028 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ current_state=started 2023-08-26T13:51:54.028 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ test started = started 2023-08-26T13:51:54.028 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ return 0 2023-08-26T13:51:54.028 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_1 2023-08-26T13:51:54.028 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local local_cluster=cluster1 2023-08-26T13:51:54.029 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster2 2023-08-26T13:51:54.029 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T13:51:54.029 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=image_1 2023-08-26T13:51:54.029 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ '[' snapshot = journal ']' 2023-08-26T13:51:54.029 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ '[' snapshot = snapshot ']' 2023-08-26T13:51:54.030 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_1 2023-08-26T13:51:54.030 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local local_cluster=cluster1 2023-08-26T13:51:54.030 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster2 2023-08-26T13:51:54.030 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T13:51:54.030 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=image_1 2023-08-26T13:51:54.030 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ mkfname cluster2-mirror-image_1.status 2023-08-26T13:51:54.031 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ echo cluster2-mirror-image_1.status 2023-08-26T13:51:54.031 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ sed -e 's|[/ ]|_|g' 2023-08-26T13:51:54.031 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local status_log=/tmp/tmp.TfDn6BcQ3N/cluster2-mirror-image_1.status 2023-08-26T13:51:54.031 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ mkfname cluster1-mirror-image_1.status 2023-08-26T13:51:54.031 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ echo cluster1-mirror-image_1.status 2023-08-26T13:51:54.031 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ sed -e 's|[/ ]|_|g' 2023-08-26T13:51:54.032 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local local_status_log=/tmp/tmp.TfDn6BcQ3N/cluster1-mirror-image_1.status 2023-08-26T13:51:54.033 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ mirror_image_snapshot cluster2 mirror image_1 2023-08-26T13:51:54.033 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster2 2023-08-26T13:51:54.033 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T13:51:54.033 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=image_1 2023-08-26T13:51:54.033 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_1 2023-08-26T13:51:54.699 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:Snapshot ID: 119 2023-08-26T13:51:54.707 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_1 /tmp/tmp.TfDn6BcQ3N/cluster2-mirror-image_1.status 2023-08-26T13:51:54.707 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster2 2023-08-26T13:51:54.707 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T13:51:54.708 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=image_1 2023-08-26T13:51:54.708 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local log=/tmp/tmp.TfDn6BcQ3N/cluster2-mirror-image_1.status 2023-08-26T13:51:54.708 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_1 --format xml 2023-08-26T13:51:54.708 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-08-26T13:51:54.750 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ xmlstarlet sel -t -v //snapshot/id 2023-08-26T13:51:54.752 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local snapshot_id=119 2023-08-26T13:51:54.752 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ true 2023-08-26T13:51:54.752 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-08-26T13:51:54.753 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ sleep 0.2 2023-08-26T13:51:54.954 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_1 /tmp/tmp.TfDn6BcQ3N/cluster1-mirror-image_1.status 2023-08-26T13:51:54.954 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster1 2023-08-26T13:51:54.954 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T13:51:54.955 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=image_1 2023-08-26T13:51:54.955 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local log=/tmp/tmp.TfDn6BcQ3N/cluster1-mirror-image_1.status 2023-08-26T13:51:54.955 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_1 --format xml 2023-08-26T13:51:54.955 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-08-26T13:51:54.998 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-08-26T13:51:54.999 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local primary_snapshot_id=68 2023-08-26T13:51:55.000 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ test 119 = 68 2023-08-26T13:51:55.000 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-08-26T13:51:55.000 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ sleep 0.4 2023-08-26T13:51:55.401 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_1 /tmp/tmp.TfDn6BcQ3N/cluster1-mirror-image_1.status 2023-08-26T13:51:55.401 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster1 2023-08-26T13:51:55.401 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T13:51:55.401 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=image_1 2023-08-26T13:51:55.402 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local log=/tmp/tmp.TfDn6BcQ3N/cluster1-mirror-image_1.status 2023-08-26T13:51:55.402 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_1 --format xml 2023-08-26T13:51:55.403 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-08-26T13:51:55.446 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-08-26T13:51:55.448 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local primary_snapshot_id=68 2023-08-26T13:51:55.448 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ test 119 = 68 2023-08-26T13:51:55.448 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-08-26T13:51:55.449 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ sleep 0.8 2023-08-26T13:51:56.250 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_1 /tmp/tmp.TfDn6BcQ3N/cluster1-mirror-image_1.status 2023-08-26T13:51:56.250 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster1 2023-08-26T13:51:56.251 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T13:51:56.251 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=image_1 2023-08-26T13:51:56.251 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local log=/tmp/tmp.TfDn6BcQ3N/cluster1-mirror-image_1.status 2023-08-26T13:51:56.251 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_1 --format xml 2023-08-26T13:51:56.251 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-08-26T13:51:56.302 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ true 2023-08-26T13:51:56.303 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-08-26T13:51:56.304 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:-:1.1: Document is empty 2023-08-26T13:51:56.304 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr: 2023-08-26T13:51:56.305 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:^ 2023-08-26T13:51:56.305 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local primary_snapshot_id= 2023-08-26T13:51:56.306 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ test 119 = '' 2023-08-26T13:51:56.306 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-08-26T13:51:56.306 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ sleep 1.6 2023-08-26T13:51:57.906 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_1 /tmp/tmp.TfDn6BcQ3N/cluster1-mirror-image_1.status 2023-08-26T13:51:57.907 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster1 2023-08-26T13:51:57.907 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T13:51:57.907 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=image_1 2023-08-26T13:51:57.907 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local log=/tmp/tmp.TfDn6BcQ3N/cluster1-mirror-image_1.status 2023-08-26T13:51:57.908 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_1 --format xml 2023-08-26T13:51:57.908 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-08-26T13:51:57.960 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-08-26T13:51:57.962 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local primary_snapshot_id=119 2023-08-26T13:51:57.962 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ test 119 = 119 2023-08-26T13:51:57.962 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ return 0 2023-08-26T13:51:57.962 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ wait_for_snap_present cluster1 mirror image_1 snap 2023-08-26T13:51:57.963 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster1 2023-08-26T13:51:57.963 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T13:51:57.963 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=image_1 2023-08-26T13:51:57.963 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local snap_name=snap 2023-08-26T13:51:57.964 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local s 2023-08-26T13:51:57.964 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2023-08-26T13:51:57.964 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ sleep 1 2023-08-26T13:51:58.962 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rbd --cluster cluster1 info mirror/image_1@snap 2023-08-26T13:51:59.007 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:rbd image 'image_1': 2023-08-26T13:51:59.007 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: size 128 MiB in 32 objects 2023-08-26T13:51:59.007 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: order 22 (4 MiB objects) 2023-08-26T13:51:59.008 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: snapshot_count: 2 2023-08-26T13:51:59.008 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: id: 10b726c08d82 2023-08-26T13:51:59.008 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: block_name_prefix: rbd_data.10b726c08d82 2023-08-26T13:51:59.008 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: format: 2 2023-08-26T13:51:59.008 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: features: layering, exclusive-lock, non-primary 2023-08-26T13:51:59.009 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: op_features: 2023-08-26T13:51:59.009 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: flags: 2023-08-26T13:51:59.009 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: create_timestamp: Sat Aug 26 13:42:54 2023 2023-08-26T13:51:59.009 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: access_timestamp: Sat Aug 26 13:42:54 2023 2023-08-26T13:51:59.010 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: modify_timestamp: Sat Aug 26 13:42:54 2023 2023-08-26T13:51:59.010 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: protected: False 2023-08-26T13:51:59.010 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: mirroring state: enabled 2023-08-26T13:51:59.010 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: mirroring mode: snapshot 2023-08-26T13:51:59.010 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: mirroring global id: cab3ca63-f126-4865-b2a3-b025783f5e8a 2023-08-26T13:51:59.011 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: mirroring primary: false 2023-08-26T13:51:59.012 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ return 0 2023-08-26T13:51:59.012 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ compare_image_snaps mirror image_1 snap 2023-08-26T13:51:59.012 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T13:51:59.012 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=image_1 2023-08-26T13:51:59.013 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local snap_name=snap 2023-08-26T13:51:59.013 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local ret=0 2023-08-26T13:51:59.013 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local rmt_export=/tmp/tmp.TfDn6BcQ3N/cluster2-mirror-image_1.export 2023-08-26T13:51:59.013 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local loc_export=/tmp/tmp.TfDn6BcQ3N/cluster1-mirror-image_1.export 2023-08-26T13:51:59.013 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rm -f /tmp/tmp.TfDn6BcQ3N/cluster2-mirror-image_1.export /tmp/tmp.TfDn6BcQ3N/cluster1-mirror-image_1.export 2023-08-26T13:51:59.014 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rbd --cluster cluster2 -p mirror export image_1@snap /tmp/tmp.TfDn6BcQ3N/cluster2-mirror-image_1.export 2023-08-26T13:51:59.190 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-08-26T13:51:59.195 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rbd --cluster cluster1 -p mirror export image_1@snap /tmp/tmp.TfDn6BcQ3N/cluster1-mirror-image_1.export 2023-08-26T13:51:59.429 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-08-26T13:51:59.434 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ cmp /tmp/tmp.TfDn6BcQ3N/cluster2-mirror-image_1.export /tmp/tmp.TfDn6BcQ3N/cluster1-mirror-image_1.export 2023-08-26T13:51:59.489 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rm -f /tmp/tmp.TfDn6BcQ3N/cluster2-mirror-image_1.export /tmp/tmp.TfDn6BcQ3N/cluster1-mirror-image_1.export 2023-08-26T13:51:59.526 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ return 0 2023-08-26T13:51:59.526 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-08-26T13:51:59.526 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ image=image_2 2023-08-26T13:51:59.526 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ create_snap cluster2 mirror image_2 snap 2023-08-26T13:51:59.527 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster2 2023-08-26T13:51:59.527 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T13:51:59.527 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=image_2 2023-08-26T13:51:59.527 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local snap_name=snap 2023-08-26T13:51:59.527 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rbd --cluster cluster2 -p mirror snap create image_2@snap --debug-rbd=20 --debug-journaler=20 2023-08-26T13:52:01.154 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ wait_for_image_replay_started cluster1 mirror image_2 2023-08-26T13:52:01.154 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster1 2023-08-26T13:52:01.154 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T13:52:01.154 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=image_2 2023-08-26T13:52:01.155 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ wait_for_image_replay_state cluster1 mirror image_2 started 2023-08-26T13:52:01.155 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster1 2023-08-26T13:52:01.155 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T13:52:01.155 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=image_2 2023-08-26T13:52:01.155 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local state=started 2023-08-26T13:52:01.156 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local s 2023-08-26T13:52:01.156 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-08-26T13:52:01.156 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ sleep 1 2023-08-26T13:52:02.155 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ test_image_replay_state cluster1 mirror image_2 started 2023-08-26T13:52:02.155 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster1 2023-08-26T13:52:02.155 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T13:52:02.156 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=image_2 2023-08-26T13:52:02.156 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local test_state=started 2023-08-26T13:52:02.156 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local status_result 2023-08-26T13:52:02.156 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local current_state=stopped 2023-08-26T13:52:02.159 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_2 2023-08-26T13:52:02.159 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local cluster_instance=cluster1 2023-08-26T13:52:02.159 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ grep -i state 2023-08-26T13:52:02.159 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ shift 2023-08-26T13:52:02.159 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local cluster=cluster1 2023-08-26T13:52:02.160 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local instance=cluster1 2023-08-26T13:52:02.160 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local loop_instance 2023-08-26T13:52:02.160 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-08-26T13:52:02.160 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ sleep 0 2023-08-26T13:52:02.160 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ '[' cluster1 '!=' cluster1 ']' 2023-08-26T13:52:02.161 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ seq 0 3 2023-08-26T13:52:02.161 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-08-26T13:52:02.161 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_2 2023-08-26T13:52:02.162 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local cluster=cluster1:0 2023-08-26T13:52:02.162 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ shift 2023-08-26T13:52:02.162 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local instance 2023-08-26T13:52:02.163 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ set_cluster_instance cluster1:0 cluster instance 2023-08-26T13:52:02.163 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local val=cluster1:0 2023-08-26T13:52:02.163 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local cluster_var_name=cluster 2023-08-26T13:52:02.163 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local instance_var_name=instance 2023-08-26T13:52:02.163 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ cluster=cluster1 2023-08-26T13:52:02.164 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ instance=0 2023-08-26T13:52:02.164 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ '[' 0 = cluster1:0 ']' 2023-08-26T13:52:02.164 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ eval cluster=cluster1 2023-08-26T13:52:02.164 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ cluster=cluster1 2023-08-26T13:52:02.164 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ eval instance=0 2023-08-26T13:52:02.165 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ instance=0 2023-08-26T13:52:02.165 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ daemon_asok_file cluster1:0 cluster1 2023-08-26T13:52:02.166 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ local local_cluster=cluster1:0 2023-08-26T13:52:02.167 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ local cluster=cluster1 2023-08-26T13:52:02.167 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ local instance 2023-08-26T13:52:02.167 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2023-08-26T13:52:02.167 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ local val=cluster1:0 2023-08-26T13:52:02.167 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ local cluster_var_name=local_cluster 2023-08-26T13:52:02.168 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ local instance_var_name=instance 2023-08-26T13:52:02.168 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ cluster=cluster1 2023-08-26T13:52:02.168 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ instance=0 2023-08-26T13:52:02.168 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ '[' 0 = cluster1:0 ']' 2023-08-26T13:52:02.168 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ eval local_cluster=cluster1 2023-08-26T13:52:02.169 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++++ local_cluster=cluster1 2023-08-26T13:52:02.169 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ eval instance=0 2023-08-26T13:52:02.169 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++++ instance=0 2023-08-26T13:52:02.169 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2023-08-26T13:52:02.181 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-08-26T13:52:02.181 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-08-26T13:52:02.181 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-08-26T13:52:02.181 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_2 2023-08-26T13:52:02.291 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:no valid command found; 2 closest matches: 2023-08-26T13:52:02.291 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:rbd mirror status mirror/image_23 2023-08-26T13:52:02.291 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:rbd mirror status mirror/image_27 2023-08-26T13:52:02.292 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:admin_socket: invalid command 2023-08-26T13:52:02.295 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-08-26T13:52:02.295 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_2 2023-08-26T13:52:02.296 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local cluster=cluster1:1 2023-08-26T13:52:02.296 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ shift 2023-08-26T13:52:02.296 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local instance 2023-08-26T13:52:02.296 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ set_cluster_instance cluster1:1 cluster instance 2023-08-26T13:52:02.296 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local val=cluster1:1 2023-08-26T13:52:02.297 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local cluster_var_name=cluster 2023-08-26T13:52:02.297 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local instance_var_name=instance 2023-08-26T13:52:02.297 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ cluster=cluster1 2023-08-26T13:52:02.297 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ instance=1 2023-08-26T13:52:02.297 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ '[' 1 = cluster1:1 ']' 2023-08-26T13:52:02.298 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ eval cluster=cluster1 2023-08-26T13:52:02.298 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ cluster=cluster1 2023-08-26T13:52:02.298 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ eval instance=1 2023-08-26T13:52:02.298 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ instance=1 2023-08-26T13:52:02.298 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ daemon_asok_file cluster1:1 cluster1 2023-08-26T13:52:02.299 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ local local_cluster=cluster1:1 2023-08-26T13:52:02.299 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ local cluster=cluster1 2023-08-26T13:52:02.299 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ local instance 2023-08-26T13:52:02.299 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2023-08-26T13:52:02.299 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ local val=cluster1:1 2023-08-26T13:52:02.299 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ local cluster_var_name=local_cluster 2023-08-26T13:52:02.300 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ local instance_var_name=instance 2023-08-26T13:52:02.300 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ cluster=cluster1 2023-08-26T13:52:02.300 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ instance=1 2023-08-26T13:52:02.300 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ '[' 1 = cluster1:1 ']' 2023-08-26T13:52:02.300 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ eval local_cluster=cluster1 2023-08-26T13:52:02.301 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++++ local_cluster=cluster1 2023-08-26T13:52:02.301 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ eval instance=1 2023-08-26T13:52:02.301 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++++ instance=1 2023-08-26T13:52:02.301 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2023-08-26T13:52:02.312 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-08-26T13:52:02.312 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-08-26T13:52:02.313 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-08-26T13:52:02.313 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_2 2023-08-26T13:52:02.426 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ return 0 2023-08-26T13:52:02.427 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ status_result=' "state": "Replaying"' 2023-08-26T13:52:02.427 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ echo ' "state": "Replaying"' 2023-08-26T13:52:02.427 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ grep -i Replaying 2023-08-26T13:52:02.428 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: "state": "Replaying" 2023-08-26T13:52:02.430 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ current_state=started 2023-08-26T13:52:02.431 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ test started = started 2023-08-26T13:52:02.431 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ return 0 2023-08-26T13:52:02.431 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_2 2023-08-26T13:52:02.431 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local local_cluster=cluster1 2023-08-26T13:52:02.431 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster2 2023-08-26T13:52:02.432 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T13:52:02.432 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=image_2 2023-08-26T13:52:02.432 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ '[' snapshot = journal ']' 2023-08-26T13:52:02.432 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ '[' snapshot = snapshot ']' 2023-08-26T13:52:02.432 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_2 2023-08-26T13:52:02.433 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local local_cluster=cluster1 2023-08-26T13:52:02.433 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster2 2023-08-26T13:52:02.433 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T13:52:02.433 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=image_2 2023-08-26T13:52:02.433 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ mkfname cluster2-mirror-image_2.status 2023-08-26T13:52:02.434 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ echo cluster2-mirror-image_2.status 2023-08-26T13:52:02.434 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ sed -e 's|[/ ]|_|g' 2023-08-26T13:52:02.434 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local status_log=/tmp/tmp.TfDn6BcQ3N/cluster2-mirror-image_2.status 2023-08-26T13:52:02.435 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ mkfname cluster1-mirror-image_2.status 2023-08-26T13:52:02.435 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ echo cluster1-mirror-image_2.status 2023-08-26T13:52:02.435 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ sed -e 's|[/ ]|_|g' 2023-08-26T13:52:02.436 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local local_status_log=/tmp/tmp.TfDn6BcQ3N/cluster1-mirror-image_2.status 2023-08-26T13:52:02.436 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ mirror_image_snapshot cluster2 mirror image_2 2023-08-26T13:52:02.437 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster2 2023-08-26T13:52:02.438 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T13:52:02.438 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=image_2 2023-08-26T13:52:02.438 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_2 2023-08-26T13:52:03.157 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:Snapshot ID: 121 2023-08-26T13:52:03.165 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_2 /tmp/tmp.TfDn6BcQ3N/cluster2-mirror-image_2.status 2023-08-26T13:52:03.165 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster2 2023-08-26T13:52:03.166 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T13:52:03.166 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=image_2 2023-08-26T13:52:03.166 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local log=/tmp/tmp.TfDn6BcQ3N/cluster2-mirror-image_2.status 2023-08-26T13:52:03.166 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_2 --format xml 2023-08-26T13:52:03.166 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-08-26T13:52:03.209 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ xmlstarlet sel -t -v //snapshot/id 2023-08-26T13:52:03.211 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local snapshot_id=121 2023-08-26T13:52:03.211 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ true 2023-08-26T13:52:03.211 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-08-26T13:52:03.211 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ sleep 0.2 2023-08-26T13:52:03.412 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_2 /tmp/tmp.TfDn6BcQ3N/cluster1-mirror-image_2.status 2023-08-26T13:52:03.412 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster1 2023-08-26T13:52:03.413 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T13:52:03.413 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=image_2 2023-08-26T13:52:03.413 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local log=/tmp/tmp.TfDn6BcQ3N/cluster1-mirror-image_2.status 2023-08-26T13:52:03.413 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_2 --format xml 2023-08-26T13:52:03.413 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-08-26T13:52:03.458 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-08-26T13:52:03.460 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local primary_snapshot_id=69 2023-08-26T13:52:03.460 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ test 121 = 69 2023-08-26T13:52:03.460 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-08-26T13:52:03.460 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ sleep 0.4 2023-08-26T13:52:03.861 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_2 /tmp/tmp.TfDn6BcQ3N/cluster1-mirror-image_2.status 2023-08-26T13:52:03.861 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster1 2023-08-26T13:52:03.861 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T13:52:03.862 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=image_2 2023-08-26T13:52:03.862 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local log=/tmp/tmp.TfDn6BcQ3N/cluster1-mirror-image_2.status 2023-08-26T13:52:03.862 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_2 --format xml 2023-08-26T13:52:03.862 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-08-26T13:52:03.907 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ true 2023-08-26T13:52:03.907 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-08-26T13:52:03.908 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:-:1.1: Document is empty 2023-08-26T13:52:03.908 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr: 2023-08-26T13:52:03.908 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:^ 2023-08-26T13:52:03.909 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local primary_snapshot_id= 2023-08-26T13:52:03.909 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ test 121 = '' 2023-08-26T13:52:03.909 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-08-26T13:52:03.910 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ sleep 0.8 2023-08-26T13:52:04.710 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_2 /tmp/tmp.TfDn6BcQ3N/cluster1-mirror-image_2.status 2023-08-26T13:52:04.710 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster1 2023-08-26T13:52:04.711 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T13:52:04.711 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=image_2 2023-08-26T13:52:04.711 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local log=/tmp/tmp.TfDn6BcQ3N/cluster1-mirror-image_2.status 2023-08-26T13:52:04.711 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_2 --format xml 2023-08-26T13:52:04.711 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-08-26T13:52:04.764 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ true 2023-08-26T13:52:04.764 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-08-26T13:52:04.765 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:-:1.1: Document is empty 2023-08-26T13:52:04.765 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr: 2023-08-26T13:52:04.766 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:^ 2023-08-26T13:52:04.766 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local primary_snapshot_id= 2023-08-26T13:52:04.767 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ test 121 = '' 2023-08-26T13:52:04.767 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-08-26T13:52:04.767 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ sleep 1.6 2023-08-26T13:52:06.367 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_2 /tmp/tmp.TfDn6BcQ3N/cluster1-mirror-image_2.status 2023-08-26T13:52:06.367 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster1 2023-08-26T13:52:06.368 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T13:52:06.368 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=image_2 2023-08-26T13:52:06.368 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local log=/tmp/tmp.TfDn6BcQ3N/cluster1-mirror-image_2.status 2023-08-26T13:52:06.368 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_2 --format xml 2023-08-26T13:52:06.368 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-08-26T13:52:06.417 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-08-26T13:52:06.420 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local primary_snapshot_id=121 2023-08-26T13:52:06.420 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ test 121 = 121 2023-08-26T13:52:06.420 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ return 0 2023-08-26T13:52:06.421 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ wait_for_snap_present cluster1 mirror image_2 snap 2023-08-26T13:52:06.421 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster1 2023-08-26T13:52:06.421 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T13:52:06.421 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=image_2 2023-08-26T13:52:06.421 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local snap_name=snap 2023-08-26T13:52:06.421 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local s 2023-08-26T13:52:06.422 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2023-08-26T13:52:06.422 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ sleep 1 2023-08-26T13:52:07.421 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rbd --cluster cluster1 info mirror/image_2@snap 2023-08-26T13:52:07.467 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:rbd image 'image_2': 2023-08-26T13:52:07.468 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: size 128 MiB in 32 objects 2023-08-26T13:52:07.468 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: order 22 (4 MiB objects) 2023-08-26T13:52:07.468 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: snapshot_count: 2 2023-08-26T13:52:07.468 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: id: 11d2d5a4c527 2023-08-26T13:52:07.468 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: block_name_prefix: rbd_data.11d2d5a4c527 2023-08-26T13:52:07.469 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: format: 2 2023-08-26T13:52:07.469 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: features: layering, exclusive-lock, non-primary 2023-08-26T13:52:07.469 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: op_features: 2023-08-26T13:52:07.469 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: flags: 2023-08-26T13:52:07.469 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: create_timestamp: Sat Aug 26 13:43:06 2023 2023-08-26T13:52:07.470 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: access_timestamp: Sat Aug 26 13:43:06 2023 2023-08-26T13:52:07.470 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: modify_timestamp: Sat Aug 26 13:43:06 2023 2023-08-26T13:52:07.470 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: protected: False 2023-08-26T13:52:07.470 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: mirroring state: enabled 2023-08-26T13:52:07.470 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: mirroring mode: snapshot 2023-08-26T13:52:07.470 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: mirroring global id: 072383ad-81e1-4717-a595-674819d66895 2023-08-26T13:52:07.471 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: mirroring primary: false 2023-08-26T13:52:07.472 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ return 0 2023-08-26T13:52:07.472 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ compare_image_snaps mirror image_2 snap 2023-08-26T13:52:07.472 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T13:52:07.473 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=image_2 2023-08-26T13:52:07.473 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local snap_name=snap 2023-08-26T13:52:07.473 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local ret=0 2023-08-26T13:52:07.473 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local rmt_export=/tmp/tmp.TfDn6BcQ3N/cluster2-mirror-image_2.export 2023-08-26T13:52:07.473 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local loc_export=/tmp/tmp.TfDn6BcQ3N/cluster1-mirror-image_2.export 2023-08-26T13:52:07.473 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rm -f /tmp/tmp.TfDn6BcQ3N/cluster2-mirror-image_2.export /tmp/tmp.TfDn6BcQ3N/cluster1-mirror-image_2.export 2023-08-26T13:52:07.474 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rbd --cluster cluster2 -p mirror export image_2@snap /tmp/tmp.TfDn6BcQ3N/cluster2-mirror-image_2.export 2023-08-26T13:52:07.650 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-08-26T13:52:07.654 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rbd --cluster cluster1 -p mirror export image_2@snap /tmp/tmp.TfDn6BcQ3N/cluster1-mirror-image_2.export 2023-08-26T13:52:07.881 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-08-26T13:52:07.886 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ cmp /tmp/tmp.TfDn6BcQ3N/cluster2-mirror-image_2.export /tmp/tmp.TfDn6BcQ3N/cluster1-mirror-image_2.export 2023-08-26T13:52:07.943 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rm -f /tmp/tmp.TfDn6BcQ3N/cluster2-mirror-image_2.export /tmp/tmp.TfDn6BcQ3N/cluster1-mirror-image_2.export 2023-08-26T13:52:07.980 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ return 0 2023-08-26T13:52:07.981 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-08-26T13:52:07.981 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ image=image_3 2023-08-26T13:52:07.981 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ create_snap cluster2 mirror image_3 snap 2023-08-26T13:52:07.982 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster2 2023-08-26T13:52:07.982 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T13:52:07.982 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=image_3 2023-08-26T13:52:07.983 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local snap_name=snap 2023-08-26T13:52:07.983 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rbd --cluster cluster2 -p mirror snap create image_3@snap --debug-rbd=20 --debug-journaler=20 2023-08-26T13:52:09.178 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ wait_for_image_replay_started cluster1 mirror image_3 2023-08-26T13:52:09.178 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster1 2023-08-26T13:52:09.179 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T13:52:09.179 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=image_3 2023-08-26T13:52:09.179 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ wait_for_image_replay_state cluster1 mirror image_3 started 2023-08-26T13:52:09.179 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster1 2023-08-26T13:52:09.180 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T13:52:09.180 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=image_3 2023-08-26T13:52:09.180 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local state=started 2023-08-26T13:52:09.180 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local s 2023-08-26T13:52:09.180 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-08-26T13:52:09.181 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ sleep 1 2023-08-26T13:52:10.179 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ test_image_replay_state cluster1 mirror image_3 started 2023-08-26T13:52:10.180 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster1 2023-08-26T13:52:10.180 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T13:52:10.180 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=image_3 2023-08-26T13:52:10.180 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local test_state=started 2023-08-26T13:52:10.181 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local status_result 2023-08-26T13:52:10.181 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local current_state=stopped 2023-08-26T13:52:10.182 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_3 2023-08-26T13:52:10.182 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local cluster_instance=cluster1 2023-08-26T13:52:10.182 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ shift 2023-08-26T13:52:10.183 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local cluster=cluster1 2023-08-26T13:52:10.183 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ grep -i state 2023-08-26T13:52:10.183 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local instance=cluster1 2023-08-26T13:52:10.183 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local loop_instance 2023-08-26T13:52:10.183 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-08-26T13:52:10.184 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ sleep 0 2023-08-26T13:52:10.184 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ '[' cluster1 '!=' cluster1 ']' 2023-08-26T13:52:10.184 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ seq 0 3 2023-08-26T13:52:10.187 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-08-26T13:52:10.187 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_3 2023-08-26T13:52:10.187 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local cluster=cluster1:0 2023-08-26T13:52:10.187 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ shift 2023-08-26T13:52:10.188 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local instance 2023-08-26T13:52:10.188 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ set_cluster_instance cluster1:0 cluster instance 2023-08-26T13:52:10.188 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local val=cluster1:0 2023-08-26T13:52:10.188 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local cluster_var_name=cluster 2023-08-26T13:52:10.188 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local instance_var_name=instance 2023-08-26T13:52:10.188 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ cluster=cluster1 2023-08-26T13:52:10.189 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ instance=0 2023-08-26T13:52:10.189 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ '[' 0 = cluster1:0 ']' 2023-08-26T13:52:10.189 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ eval cluster=cluster1 2023-08-26T13:52:10.189 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ cluster=cluster1 2023-08-26T13:52:10.189 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ eval instance=0 2023-08-26T13:52:10.190 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ instance=0 2023-08-26T13:52:10.190 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ daemon_asok_file cluster1:0 cluster1 2023-08-26T13:52:10.190 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ local local_cluster=cluster1:0 2023-08-26T13:52:10.190 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ local cluster=cluster1 2023-08-26T13:52:10.190 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ local instance 2023-08-26T13:52:10.191 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2023-08-26T13:52:10.191 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ local val=cluster1:0 2023-08-26T13:52:10.191 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ local cluster_var_name=local_cluster 2023-08-26T13:52:10.191 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ local instance_var_name=instance 2023-08-26T13:52:10.191 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ cluster=cluster1 2023-08-26T13:52:10.192 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ instance=0 2023-08-26T13:52:10.192 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ '[' 0 = cluster1:0 ']' 2023-08-26T13:52:10.192 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ eval local_cluster=cluster1 2023-08-26T13:52:10.192 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++++ local_cluster=cluster1 2023-08-26T13:52:10.192 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ eval instance=0 2023-08-26T13:52:10.192 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++++ instance=0 2023-08-26T13:52:10.193 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2023-08-26T13:52:10.206 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-08-26T13:52:10.206 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-08-26T13:52:10.206 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-08-26T13:52:10.206 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_3 2023-08-26T13:52:10.315 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:no valid command found; 3 closest matches: 2023-08-26T13:52:10.316 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:rbd mirror status mirror/image_30 2023-08-26T13:52:10.316 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:rbd mirror status mirror/image_32 2023-08-26T13:52:10.316 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:rbd mirror status mirror/image_36 2023-08-26T13:52:10.316 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:admin_socket: invalid command 2023-08-26T13:52:10.319 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-08-26T13:52:10.319 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_3 2023-08-26T13:52:10.319 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local cluster=cluster1:1 2023-08-26T13:52:10.319 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ shift 2023-08-26T13:52:10.320 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local instance 2023-08-26T13:52:10.320 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ set_cluster_instance cluster1:1 cluster instance 2023-08-26T13:52:10.320 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local val=cluster1:1 2023-08-26T13:52:10.320 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local cluster_var_name=cluster 2023-08-26T13:52:10.320 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local instance_var_name=instance 2023-08-26T13:52:10.321 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ cluster=cluster1 2023-08-26T13:52:10.321 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ instance=1 2023-08-26T13:52:10.321 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ '[' 1 = cluster1:1 ']' 2023-08-26T13:52:10.321 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ eval cluster=cluster1 2023-08-26T13:52:10.321 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ cluster=cluster1 2023-08-26T13:52:10.322 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ eval instance=1 2023-08-26T13:52:10.322 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ instance=1 2023-08-26T13:52:10.322 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ daemon_asok_file cluster1:1 cluster1 2023-08-26T13:52:10.322 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ local local_cluster=cluster1:1 2023-08-26T13:52:10.322 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ local cluster=cluster1 2023-08-26T13:52:10.322 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ local instance 2023-08-26T13:52:10.323 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2023-08-26T13:52:10.323 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ local val=cluster1:1 2023-08-26T13:52:10.323 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ local cluster_var_name=local_cluster 2023-08-26T13:52:10.323 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ local instance_var_name=instance 2023-08-26T13:52:10.323 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ cluster=cluster1 2023-08-26T13:52:10.324 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ instance=1 2023-08-26T13:52:10.324 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ '[' 1 = cluster1:1 ']' 2023-08-26T13:52:10.324 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ eval local_cluster=cluster1 2023-08-26T13:52:10.324 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++++ local_cluster=cluster1 2023-08-26T13:52:10.324 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ eval instance=1 2023-08-26T13:52:10.325 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++++ instance=1 2023-08-26T13:52:10.325 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2023-08-26T13:52:10.337 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-08-26T13:52:10.337 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-08-26T13:52:10.338 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-08-26T13:52:10.338 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_3 2023-08-26T13:52:10.443 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:no valid command found; 4 closest matches: 2023-08-26T13:52:10.443 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:rbd mirror status mirror/image_31 2023-08-26T13:52:10.444 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:rbd mirror status mirror/image_34 2023-08-26T13:52:10.444 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:rbd mirror status mirror/image_35 2023-08-26T13:52:10.444 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:rbd mirror status mirror/image_39 2023-08-26T13:52:10.444 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:admin_socket: invalid command 2023-08-26T13:52:10.447 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-08-26T13:52:10.447 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/image_3 2023-08-26T13:52:10.447 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local cluster=cluster1:2 2023-08-26T13:52:10.448 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ shift 2023-08-26T13:52:10.448 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local instance 2023-08-26T13:52:10.448 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ set_cluster_instance cluster1:2 cluster instance 2023-08-26T13:52:10.448 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local val=cluster1:2 2023-08-26T13:52:10.448 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local cluster_var_name=cluster 2023-08-26T13:52:10.449 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local instance_var_name=instance 2023-08-26T13:52:10.449 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ cluster=cluster1 2023-08-26T13:52:10.449 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ instance=2 2023-08-26T13:52:10.449 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ '[' 2 = cluster1:2 ']' 2023-08-26T13:52:10.449 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ eval cluster=cluster1 2023-08-26T13:52:10.450 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ cluster=cluster1 2023-08-26T13:52:10.450 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ eval instance=2 2023-08-26T13:52:10.450 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ instance=2 2023-08-26T13:52:10.450 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ daemon_asok_file cluster1:2 cluster1 2023-08-26T13:52:10.450 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ local local_cluster=cluster1:2 2023-08-26T13:52:10.450 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ local cluster=cluster1 2023-08-26T13:52:10.451 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ local instance 2023-08-26T13:52:10.451 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2023-08-26T13:52:10.451 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ local val=cluster1:2 2023-08-26T13:52:10.451 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ local cluster_var_name=local_cluster 2023-08-26T13:52:10.451 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ local instance_var_name=instance 2023-08-26T13:52:10.452 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ cluster=cluster1 2023-08-26T13:52:10.452 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ instance=2 2023-08-26T13:52:10.452 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ '[' 2 = cluster1:2 ']' 2023-08-26T13:52:10.452 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ eval local_cluster=cluster1 2023-08-26T13:52:10.452 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++++ local_cluster=cluster1 2023-08-26T13:52:10.452 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ eval instance=2 2023-08-26T13:52:10.453 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++++ instance=2 2023-08-26T13:52:10.453 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2023-08-26T13:52:10.465 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-08-26T13:52:10.465 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-08-26T13:52:10.465 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-08-26T13:52:10.466 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/image_3 2023-08-26T13:52:10.578 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ return 0 2023-08-26T13:52:10.578 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ status_result=' "state": "Replaying"' 2023-08-26T13:52:10.579 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ echo ' "state": "Replaying"' 2023-08-26T13:52:10.579 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ grep -i Replaying 2023-08-26T13:52:10.579 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: "state": "Replaying" 2023-08-26T13:52:10.582 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ current_state=started 2023-08-26T13:52:10.582 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ test started = started 2023-08-26T13:52:10.582 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ return 0 2023-08-26T13:52:10.583 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_3 2023-08-26T13:52:10.584 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local local_cluster=cluster1 2023-08-26T13:52:10.584 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster2 2023-08-26T13:52:10.584 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T13:52:10.584 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=image_3 2023-08-26T13:52:10.585 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ '[' snapshot = journal ']' 2023-08-26T13:52:10.585 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ '[' snapshot = snapshot ']' 2023-08-26T13:52:10.585 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_3 2023-08-26T13:52:10.585 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local local_cluster=cluster1 2023-08-26T13:52:10.585 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster2 2023-08-26T13:52:10.585 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T13:52:10.586 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=image_3 2023-08-26T13:52:10.586 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ mkfname cluster2-mirror-image_3.status 2023-08-26T13:52:10.586 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ echo cluster2-mirror-image_3.status 2023-08-26T13:52:10.586 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ sed -e 's|[/ ]|_|g' 2023-08-26T13:52:10.586 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local status_log=/tmp/tmp.TfDn6BcQ3N/cluster2-mirror-image_3.status 2023-08-26T13:52:10.587 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ mkfname cluster1-mirror-image_3.status 2023-08-26T13:52:10.587 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ echo cluster1-mirror-image_3.status 2023-08-26T13:52:10.587 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ sed -e 's|[/ ]|_|g' 2023-08-26T13:52:10.588 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local local_status_log=/tmp/tmp.TfDn6BcQ3N/cluster1-mirror-image_3.status 2023-08-26T13:52:10.588 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ mirror_image_snapshot cluster2 mirror image_3 2023-08-26T13:52:10.589 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster2 2023-08-26T13:52:10.589 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T13:52:10.589 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=image_3 2023-08-26T13:52:10.589 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_3 2023-08-26T13:52:11.181 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:Snapshot ID: 123 2023-08-26T13:52:11.189 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_3 /tmp/tmp.TfDn6BcQ3N/cluster2-mirror-image_3.status 2023-08-26T13:52:11.189 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster2 2023-08-26T13:52:11.190 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T13:52:11.190 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=image_3 2023-08-26T13:52:11.190 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local log=/tmp/tmp.TfDn6BcQ3N/cluster2-mirror-image_3.status 2023-08-26T13:52:11.190 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_3 --format xml 2023-08-26T13:52:11.190 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-08-26T13:52:11.232 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ xmlstarlet sel -t -v //snapshot/id 2023-08-26T13:52:11.234 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local snapshot_id=123 2023-08-26T13:52:11.234 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ true 2023-08-26T13:52:11.234 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-08-26T13:52:11.234 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ sleep 0.2 2023-08-26T13:52:11.435 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_3 /tmp/tmp.TfDn6BcQ3N/cluster1-mirror-image_3.status 2023-08-26T13:52:11.435 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster1 2023-08-26T13:52:11.436 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T13:52:11.436 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=image_3 2023-08-26T13:52:11.436 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local log=/tmp/tmp.TfDn6BcQ3N/cluster1-mirror-image_3.status 2023-08-26T13:52:11.436 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_3 --format xml 2023-08-26T13:52:11.436 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-08-26T13:52:11.483 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ true 2023-08-26T13:52:11.484 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-08-26T13:52:11.485 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:-:1.1: Document is empty 2023-08-26T13:52:11.485 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr: 2023-08-26T13:52:11.485 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:^ 2023-08-26T13:52:11.486 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local primary_snapshot_id= 2023-08-26T13:52:11.486 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ test 123 = '' 2023-08-26T13:52:11.486 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-08-26T13:52:11.487 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ sleep 0.4 2023-08-26T13:52:11.887 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_3 /tmp/tmp.TfDn6BcQ3N/cluster1-mirror-image_3.status 2023-08-26T13:52:11.887 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster1 2023-08-26T13:52:11.887 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T13:52:11.888 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=image_3 2023-08-26T13:52:11.888 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local log=/tmp/tmp.TfDn6BcQ3N/cluster1-mirror-image_3.status 2023-08-26T13:52:11.888 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_3 --format xml 2023-08-26T13:52:11.889 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-08-26T13:52:11.933 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ true 2023-08-26T13:52:11.933 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-08-26T13:52:11.935 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:-:1.1: Document is empty 2023-08-26T13:52:11.935 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr: 2023-08-26T13:52:11.935 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:^ 2023-08-26T13:52:11.935 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local primary_snapshot_id= 2023-08-26T13:52:11.936 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ test 123 = '' 2023-08-26T13:52:11.936 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-08-26T13:52:11.936 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ sleep 0.8 2023-08-26T13:52:12.736 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_3 /tmp/tmp.TfDn6BcQ3N/cluster1-mirror-image_3.status 2023-08-26T13:52:12.737 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster1 2023-08-26T13:52:12.737 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T13:52:12.737 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=image_3 2023-08-26T13:52:12.737 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local log=/tmp/tmp.TfDn6BcQ3N/cluster1-mirror-image_3.status 2023-08-26T13:52:12.737 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_3 --format xml 2023-08-26T13:52:12.738 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-08-26T13:52:12.782 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-08-26T13:52:12.785 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local primary_snapshot_id=123 2023-08-26T13:52:12.786 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ test 123 = 123 2023-08-26T13:52:12.786 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ return 0 2023-08-26T13:52:12.786 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ wait_for_snap_present cluster1 mirror image_3 snap 2023-08-26T13:52:12.787 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster1 2023-08-26T13:52:12.787 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T13:52:12.787 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=image_3 2023-08-26T13:52:12.788 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local snap_name=snap 2023-08-26T13:52:12.788 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local s 2023-08-26T13:52:12.788 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2023-08-26T13:52:12.788 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ sleep 1 2023-08-26T13:52:13.785 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rbd --cluster cluster1 info mirror/image_3@snap 2023-08-26T13:52:13.828 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:rbd image 'image_3': 2023-08-26T13:52:13.829 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: size 128 MiB in 32 objects 2023-08-26T13:52:13.829 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: order 22 (4 MiB objects) 2023-08-26T13:52:13.829 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: snapshot_count: 2 2023-08-26T13:52:13.829 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: id: 122b5e59fcf5 2023-08-26T13:52:13.829 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: block_name_prefix: rbd_data.122b5e59fcf5 2023-08-26T13:52:13.830 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: format: 2 2023-08-26T13:52:13.830 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: features: layering, exclusive-lock, non-primary 2023-08-26T13:52:13.830 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: op_features: 2023-08-26T13:52:13.831 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: flags: 2023-08-26T13:52:13.831 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: create_timestamp: Sat Aug 26 13:43:12 2023 2023-08-26T13:52:13.831 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: access_timestamp: Sat Aug 26 13:43:12 2023 2023-08-26T13:52:13.831 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: modify_timestamp: Sat Aug 26 13:43:12 2023 2023-08-26T13:52:13.832 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: protected: False 2023-08-26T13:52:13.832 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: mirroring state: enabled 2023-08-26T13:52:13.832 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: mirroring mode: snapshot 2023-08-26T13:52:13.832 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: mirroring global id: 339416f3-5c35-427b-8a40-65d1d209d742 2023-08-26T13:52:13.832 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: mirroring primary: false 2023-08-26T13:52:13.834 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ return 0 2023-08-26T13:52:13.834 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ compare_image_snaps mirror image_3 snap 2023-08-26T13:52:13.834 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T13:52:13.834 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=image_3 2023-08-26T13:52:13.835 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local snap_name=snap 2023-08-26T13:52:13.835 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local ret=0 2023-08-26T13:52:13.835 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local rmt_export=/tmp/tmp.TfDn6BcQ3N/cluster2-mirror-image_3.export 2023-08-26T13:52:13.835 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local loc_export=/tmp/tmp.TfDn6BcQ3N/cluster1-mirror-image_3.export 2023-08-26T13:52:13.835 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rm -f /tmp/tmp.TfDn6BcQ3N/cluster2-mirror-image_3.export /tmp/tmp.TfDn6BcQ3N/cluster1-mirror-image_3.export 2023-08-26T13:52:13.836 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rbd --cluster cluster2 -p mirror export image_3@snap /tmp/tmp.TfDn6BcQ3N/cluster2-mirror-image_3.export 2023-08-26T13:52:14.015 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-08-26T13:52:14.020 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rbd --cluster cluster1 -p mirror export image_3@snap /tmp/tmp.TfDn6BcQ3N/cluster1-mirror-image_3.export 2023-08-26T13:52:14.241 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-08-26T13:52:14.248 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ cmp /tmp/tmp.TfDn6BcQ3N/cluster2-mirror-image_3.export /tmp/tmp.TfDn6BcQ3N/cluster1-mirror-image_3.export 2023-08-26T13:52:14.303 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rm -f /tmp/tmp.TfDn6BcQ3N/cluster2-mirror-image_3.export /tmp/tmp.TfDn6BcQ3N/cluster1-mirror-image_3.export 2023-08-26T13:52:14.340 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ return 0 2023-08-26T13:52:14.341 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-08-26T13:52:14.341 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ image=image_4 2023-08-26T13:52:14.341 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ create_snap cluster2 mirror image_4 snap 2023-08-26T13:52:14.341 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster2 2023-08-26T13:52:14.341 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T13:52:14.342 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=image_4 2023-08-26T13:52:14.342 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local snap_name=snap 2023-08-26T13:52:14.342 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rbd --cluster cluster2 -p mirror snap create image_4@snap --debug-rbd=20 --debug-journaler=20 2023-08-26T13:52:16.197 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ wait_for_image_replay_started cluster1 mirror image_4 2023-08-26T13:52:16.198 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster1 2023-08-26T13:52:16.198 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T13:52:16.198 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=image_4 2023-08-26T13:52:16.198 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ wait_for_image_replay_state cluster1 mirror image_4 started 2023-08-26T13:52:16.198 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster1 2023-08-26T13:52:16.199 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T13:52:16.199 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=image_4 2023-08-26T13:52:16.199 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local state=started 2023-08-26T13:52:16.199 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local s 2023-08-26T13:52:16.200 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-08-26T13:52:16.200 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ sleep 1 2023-08-26T13:52:17.198 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ test_image_replay_state cluster1 mirror image_4 started 2023-08-26T13:52:17.199 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster1 2023-08-26T13:52:17.199 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T13:52:17.199 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=image_4 2023-08-26T13:52:17.199 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local test_state=started 2023-08-26T13:52:17.200 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local status_result 2023-08-26T13:52:17.200 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local current_state=stopped 2023-08-26T13:52:17.201 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_4 2023-08-26T13:52:17.201 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local cluster_instance=cluster1 2023-08-26T13:52:17.201 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ shift 2023-08-26T13:52:17.202 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local cluster=cluster1 2023-08-26T13:52:17.202 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local instance=cluster1 2023-08-26T13:52:17.202 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ grep -i state 2023-08-26T13:52:17.202 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local loop_instance 2023-08-26T13:52:17.202 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-08-26T13:52:17.203 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ sleep 0 2023-08-26T13:52:17.203 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ '[' cluster1 '!=' cluster1 ']' 2023-08-26T13:52:17.203 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ seq 0 3 2023-08-26T13:52:17.206 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-08-26T13:52:17.206 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_4 2023-08-26T13:52:17.206 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local cluster=cluster1:0 2023-08-26T13:52:17.206 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ shift 2023-08-26T13:52:17.207 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local instance 2023-08-26T13:52:17.207 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ set_cluster_instance cluster1:0 cluster instance 2023-08-26T13:52:17.207 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local val=cluster1:0 2023-08-26T13:52:17.207 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local cluster_var_name=cluster 2023-08-26T13:52:17.207 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local instance_var_name=instance 2023-08-26T13:52:17.208 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ cluster=cluster1 2023-08-26T13:52:17.208 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ instance=0 2023-08-26T13:52:17.208 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ '[' 0 = cluster1:0 ']' 2023-08-26T13:52:17.209 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ eval cluster=cluster1 2023-08-26T13:52:17.209 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ cluster=cluster1 2023-08-26T13:52:17.209 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ eval instance=0 2023-08-26T13:52:17.209 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ instance=0 2023-08-26T13:52:17.210 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ daemon_asok_file cluster1:0 cluster1 2023-08-26T13:52:17.211 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ local local_cluster=cluster1:0 2023-08-26T13:52:17.211 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ local cluster=cluster1 2023-08-26T13:52:17.211 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ local instance 2023-08-26T13:52:17.212 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2023-08-26T13:52:17.212 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ local val=cluster1:0 2023-08-26T13:52:17.212 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ local cluster_var_name=local_cluster 2023-08-26T13:52:17.213 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ local instance_var_name=instance 2023-08-26T13:52:17.213 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ cluster=cluster1 2023-08-26T13:52:17.213 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ instance=0 2023-08-26T13:52:17.214 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ '[' 0 = cluster1:0 ']' 2023-08-26T13:52:17.214 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ eval local_cluster=cluster1 2023-08-26T13:52:17.215 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++++ local_cluster=cluster1 2023-08-26T13:52:17.215 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ eval instance=0 2023-08-26T13:52:17.216 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++++ instance=0 2023-08-26T13:52:17.216 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2023-08-26T13:52:17.225 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-08-26T13:52:17.226 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-08-26T13:52:17.226 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-08-26T13:52:17.226 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_4 2023-08-26T13:52:17.336 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:no valid command found; 3 closest matches: 2023-08-26T13:52:17.336 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:rbd mirror status mirror/image_40 2023-08-26T13:52:17.336 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:rbd mirror status mirror/image_44 2023-08-26T13:52:17.337 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:rbd mirror status mirror/image_48 2023-08-26T13:52:17.337 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:admin_socket: invalid command 2023-08-26T13:52:17.340 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-08-26T13:52:17.340 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_4 2023-08-26T13:52:17.341 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local cluster=cluster1:1 2023-08-26T13:52:17.341 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ shift 2023-08-26T13:52:17.341 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local instance 2023-08-26T13:52:17.341 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ set_cluster_instance cluster1:1 cluster instance 2023-08-26T13:52:17.341 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local val=cluster1:1 2023-08-26T13:52:17.342 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local cluster_var_name=cluster 2023-08-26T13:52:17.342 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local instance_var_name=instance 2023-08-26T13:52:17.342 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ cluster=cluster1 2023-08-26T13:52:17.342 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ instance=1 2023-08-26T13:52:17.342 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ '[' 1 = cluster1:1 ']' 2023-08-26T13:52:17.343 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ eval cluster=cluster1 2023-08-26T13:52:17.343 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ cluster=cluster1 2023-08-26T13:52:17.343 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ eval instance=1 2023-08-26T13:52:17.343 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ instance=1 2023-08-26T13:52:17.343 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ daemon_asok_file cluster1:1 cluster1 2023-08-26T13:52:17.344 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ local local_cluster=cluster1:1 2023-08-26T13:52:17.344 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ local cluster=cluster1 2023-08-26T13:52:17.344 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ local instance 2023-08-26T13:52:17.344 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2023-08-26T13:52:17.344 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ local val=cluster1:1 2023-08-26T13:52:17.345 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ local cluster_var_name=local_cluster 2023-08-26T13:52:17.345 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ local instance_var_name=instance 2023-08-26T13:52:17.345 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ cluster=cluster1 2023-08-26T13:52:17.345 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ instance=1 2023-08-26T13:52:17.346 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ '[' 1 = cluster1:1 ']' 2023-08-26T13:52:17.346 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ eval local_cluster=cluster1 2023-08-26T13:52:17.346 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++++ local_cluster=cluster1 2023-08-26T13:52:17.346 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ eval instance=1 2023-08-26T13:52:17.347 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++++ instance=1 2023-08-26T13:52:17.348 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2023-08-26T13:52:17.359 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-08-26T13:52:17.359 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-08-26T13:52:17.359 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-08-26T13:52:17.360 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_4 2023-08-26T13:52:17.464 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:no valid command found; 2 closest matches: 2023-08-26T13:52:17.464 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:rbd mirror status mirror/image_43 2023-08-26T13:52:17.464 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:rbd mirror status mirror/image_47 2023-08-26T13:52:17.464 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:admin_socket: invalid command 2023-08-26T13:52:17.468 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-08-26T13:52:17.468 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/image_4 2023-08-26T13:52:17.468 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local cluster=cluster1:2 2023-08-26T13:52:17.469 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ shift 2023-08-26T13:52:17.469 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local instance 2023-08-26T13:52:17.469 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ set_cluster_instance cluster1:2 cluster instance 2023-08-26T13:52:17.469 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local val=cluster1:2 2023-08-26T13:52:17.469 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local cluster_var_name=cluster 2023-08-26T13:52:17.470 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local instance_var_name=instance 2023-08-26T13:52:17.470 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ cluster=cluster1 2023-08-26T13:52:17.470 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ instance=2 2023-08-26T13:52:17.470 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ '[' 2 = cluster1:2 ']' 2023-08-26T13:52:17.470 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ eval cluster=cluster1 2023-08-26T13:52:17.470 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ cluster=cluster1 2023-08-26T13:52:17.471 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ eval instance=2 2023-08-26T13:52:17.472 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ instance=2 2023-08-26T13:52:17.472 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ daemon_asok_file cluster1:2 cluster1 2023-08-26T13:52:17.472 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ local local_cluster=cluster1:2 2023-08-26T13:52:17.473 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ local cluster=cluster1 2023-08-26T13:52:17.473 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ local instance 2023-08-26T13:52:17.473 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2023-08-26T13:52:17.474 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ local val=cluster1:2 2023-08-26T13:52:17.474 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ local cluster_var_name=local_cluster 2023-08-26T13:52:17.474 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ local instance_var_name=instance 2023-08-26T13:52:17.475 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ cluster=cluster1 2023-08-26T13:52:17.475 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ instance=2 2023-08-26T13:52:17.475 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ '[' 2 = cluster1:2 ']' 2023-08-26T13:52:17.476 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ eval local_cluster=cluster1 2023-08-26T13:52:17.476 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++++ local_cluster=cluster1 2023-08-26T13:52:17.476 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ eval instance=2 2023-08-26T13:52:17.476 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++++ instance=2 2023-08-26T13:52:17.477 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2023-08-26T13:52:17.487 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-08-26T13:52:17.488 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-08-26T13:52:17.488 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-08-26T13:52:17.489 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/image_4 2023-08-26T13:52:17.599 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ return 0 2023-08-26T13:52:17.600 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ status_result=' "state": "Replaying"' 2023-08-26T13:52:17.600 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ echo ' "state": "Replaying"' 2023-08-26T13:52:17.601 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ grep -i Replaying 2023-08-26T13:52:17.601 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: "state": "Replaying" 2023-08-26T13:52:17.603 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ current_state=started 2023-08-26T13:52:17.603 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ test started = started 2023-08-26T13:52:17.603 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ return 0 2023-08-26T13:52:17.604 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_4 2023-08-26T13:52:17.604 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local local_cluster=cluster1 2023-08-26T13:52:17.604 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster2 2023-08-26T13:52:17.604 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T13:52:17.604 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=image_4 2023-08-26T13:52:17.605 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ '[' snapshot = journal ']' 2023-08-26T13:52:17.605 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ '[' snapshot = snapshot ']' 2023-08-26T13:52:17.605 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_4 2023-08-26T13:52:17.605 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local local_cluster=cluster1 2023-08-26T13:52:17.605 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster2 2023-08-26T13:52:17.605 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T13:52:17.606 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=image_4 2023-08-26T13:52:17.606 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ mkfname cluster2-mirror-image_4.status 2023-08-26T13:52:17.606 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ echo cluster2-mirror-image_4.status 2023-08-26T13:52:17.606 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ sed -e 's|[/ ]|_|g' 2023-08-26T13:52:17.606 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local status_log=/tmp/tmp.TfDn6BcQ3N/cluster2-mirror-image_4.status 2023-08-26T13:52:17.607 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ mkfname cluster1-mirror-image_4.status 2023-08-26T13:52:17.607 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ echo cluster1-mirror-image_4.status 2023-08-26T13:52:17.607 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ sed -e 's|[/ ]|_|g' 2023-08-26T13:52:17.608 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local local_status_log=/tmp/tmp.TfDn6BcQ3N/cluster1-mirror-image_4.status 2023-08-26T13:52:17.608 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ mirror_image_snapshot cluster2 mirror image_4 2023-08-26T13:52:17.608 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster2 2023-08-26T13:52:17.608 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T13:52:17.609 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=image_4 2023-08-26T13:52:17.609 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_4 2023-08-26T13:52:18.200 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:Snapshot ID: 125 2023-08-26T13:52:18.209 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_4 /tmp/tmp.TfDn6BcQ3N/cluster2-mirror-image_4.status 2023-08-26T13:52:18.209 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster2 2023-08-26T13:52:18.210 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T13:52:18.210 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=image_4 2023-08-26T13:52:18.210 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local log=/tmp/tmp.TfDn6BcQ3N/cluster2-mirror-image_4.status 2023-08-26T13:52:18.210 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_4 --format xml 2023-08-26T13:52:18.211 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-08-26T13:52:18.252 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ xmlstarlet sel -t -v //snapshot/id 2023-08-26T13:52:18.254 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local snapshot_id=125 2023-08-26T13:52:18.254 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ true 2023-08-26T13:52:18.254 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-08-26T13:52:18.254 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ sleep 0.2 2023-08-26T13:52:18.455 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_4 /tmp/tmp.TfDn6BcQ3N/cluster1-mirror-image_4.status 2023-08-26T13:52:18.455 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster1 2023-08-26T13:52:18.456 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T13:52:18.456 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=image_4 2023-08-26T13:52:18.456 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local log=/tmp/tmp.TfDn6BcQ3N/cluster1-mirror-image_4.status 2023-08-26T13:52:18.456 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_4 --format xml 2023-08-26T13:52:18.456 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-08-26T13:52:18.501 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ true 2023-08-26T13:52:18.502 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-08-26T13:52:18.503 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:-:1.1: Document is empty 2023-08-26T13:52:18.504 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr: 2023-08-26T13:52:18.504 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:^ 2023-08-26T13:52:18.504 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local primary_snapshot_id= 2023-08-26T13:52:18.504 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ test 125 = '' 2023-08-26T13:52:18.505 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-08-26T13:52:18.505 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ sleep 0.4 2023-08-26T13:52:18.905 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_4 /tmp/tmp.TfDn6BcQ3N/cluster1-mirror-image_4.status 2023-08-26T13:52:18.905 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster1 2023-08-26T13:52:18.905 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T13:52:18.905 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=image_4 2023-08-26T13:52:18.906 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local log=/tmp/tmp.TfDn6BcQ3N/cluster1-mirror-image_4.status 2023-08-26T13:52:18.906 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_4 --format xml 2023-08-26T13:52:18.906 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-08-26T13:52:18.949 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ true 2023-08-26T13:52:18.950 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-08-26T13:52:18.950 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:-:1.1: Document is empty 2023-08-26T13:52:18.951 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr: 2023-08-26T13:52:18.951 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:^ 2023-08-26T13:52:18.951 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local primary_snapshot_id= 2023-08-26T13:52:18.952 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ test 125 = '' 2023-08-26T13:52:18.952 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-08-26T13:52:18.952 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ sleep 0.8 2023-08-26T13:52:19.752 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_4 /tmp/tmp.TfDn6BcQ3N/cluster1-mirror-image_4.status 2023-08-26T13:52:19.753 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster1 2023-08-26T13:52:19.753 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T13:52:19.753 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=image_4 2023-08-26T13:52:19.754 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local log=/tmp/tmp.TfDn6BcQ3N/cluster1-mirror-image_4.status 2023-08-26T13:52:19.754 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_4 --format xml 2023-08-26T13:52:19.754 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-08-26T13:52:19.798 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-08-26T13:52:19.800 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local primary_snapshot_id=125 2023-08-26T13:52:19.801 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ test 125 = 125 2023-08-26T13:52:19.801 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ return 0 2023-08-26T13:52:19.802 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ wait_for_snap_present cluster1 mirror image_4 snap 2023-08-26T13:52:19.802 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster1 2023-08-26T13:52:19.802 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T13:52:19.802 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=image_4 2023-08-26T13:52:19.803 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local snap_name=snap 2023-08-26T13:52:19.803 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local s 2023-08-26T13:52:19.803 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2023-08-26T13:52:19.803 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ sleep 1 2023-08-26T13:52:20.801 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rbd --cluster cluster1 info mirror/image_4@snap 2023-08-26T13:52:20.847 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:rbd image 'image_4': 2023-08-26T13:52:20.847 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: size 128 MiB in 32 objects 2023-08-26T13:52:20.847 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: order 22 (4 MiB objects) 2023-08-26T13:52:20.847 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: snapshot_count: 2 2023-08-26T13:52:20.848 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: id: 12a08ac1fddc 2023-08-26T13:52:20.848 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: block_name_prefix: rbd_data.12a08ac1fddc 2023-08-26T13:52:20.848 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: format: 2 2023-08-26T13:52:20.848 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: features: layering, exclusive-lock, non-primary 2023-08-26T13:52:20.848 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: op_features: 2023-08-26T13:52:20.849 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: flags: 2023-08-26T13:52:20.849 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: create_timestamp: Sat Aug 26 13:43:18 2023 2023-08-26T13:52:20.849 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: access_timestamp: Sat Aug 26 13:43:18 2023 2023-08-26T13:52:20.849 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: modify_timestamp: Sat Aug 26 13:43:18 2023 2023-08-26T13:52:20.849 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: protected: False 2023-08-26T13:52:20.850 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: mirroring state: enabled 2023-08-26T13:52:20.850 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: mirroring mode: snapshot 2023-08-26T13:52:20.850 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: mirroring global id: 2f906edf-8e50-48f7-a42e-a4055f204c44 2023-08-26T13:52:20.850 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: mirroring primary: false 2023-08-26T13:52:20.851 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ return 0 2023-08-26T13:52:20.851 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ compare_image_snaps mirror image_4 snap 2023-08-26T13:52:20.852 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T13:52:20.852 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=image_4 2023-08-26T13:52:20.852 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local snap_name=snap 2023-08-26T13:52:20.852 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local ret=0 2023-08-26T13:52:20.852 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local rmt_export=/tmp/tmp.TfDn6BcQ3N/cluster2-mirror-image_4.export 2023-08-26T13:52:20.853 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local loc_export=/tmp/tmp.TfDn6BcQ3N/cluster1-mirror-image_4.export 2023-08-26T13:52:20.853 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rm -f /tmp/tmp.TfDn6BcQ3N/cluster2-mirror-image_4.export /tmp/tmp.TfDn6BcQ3N/cluster1-mirror-image_4.export 2023-08-26T13:52:20.853 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rbd --cluster cluster2 -p mirror export image_4@snap /tmp/tmp.TfDn6BcQ3N/cluster2-mirror-image_4.export 2023-08-26T13:52:21.041 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-08-26T13:52:21.048 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rbd --cluster cluster1 -p mirror export image_4@snap /tmp/tmp.TfDn6BcQ3N/cluster1-mirror-image_4.export 2023-08-26T13:52:21.263 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-08-26T13:52:21.268 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ cmp /tmp/tmp.TfDn6BcQ3N/cluster2-mirror-image_4.export /tmp/tmp.TfDn6BcQ3N/cluster1-mirror-image_4.export 2023-08-26T13:52:21.324 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rm -f /tmp/tmp.TfDn6BcQ3N/cluster2-mirror-image_4.export /tmp/tmp.TfDn6BcQ3N/cluster1-mirror-image_4.export 2023-08-26T13:52:21.361 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ return 0 2023-08-26T13:52:21.361 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-08-26T13:52:21.361 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ image=image_5 2023-08-26T13:52:21.361 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ create_snap cluster2 mirror image_5 snap 2023-08-26T13:52:21.362 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster2 2023-08-26T13:52:21.362 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T13:52:21.362 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=image_5 2023-08-26T13:52:21.362 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local snap_name=snap 2023-08-26T13:52:21.362 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rbd --cluster cluster2 -p mirror snap create image_5@snap --debug-rbd=20 --debug-journaler=20 2023-08-26T13:52:23.211 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ wait_for_image_replay_started cluster1 mirror image_5 2023-08-26T13:52:23.211 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster1 2023-08-26T13:52:23.211 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T13:52:23.211 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=image_5 2023-08-26T13:52:23.212 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ wait_for_image_replay_state cluster1 mirror image_5 started 2023-08-26T13:52:23.212 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster1 2023-08-26T13:52:23.212 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T13:52:23.212 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=image_5 2023-08-26T13:52:23.212 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local state=started 2023-08-26T13:52:23.213 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local s 2023-08-26T13:52:23.213 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-08-26T13:52:23.213 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ sleep 1 2023-08-26T13:52:24.212 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ test_image_replay_state cluster1 mirror image_5 started 2023-08-26T13:52:24.212 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster1 2023-08-26T13:52:24.212 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T13:52:24.212 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=image_5 2023-08-26T13:52:24.213 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local test_state=started 2023-08-26T13:52:24.213 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local status_result 2023-08-26T13:52:24.213 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local current_state=stopped 2023-08-26T13:52:24.214 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_5 2023-08-26T13:52:24.215 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ grep -i state 2023-08-26T13:52:24.215 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local cluster_instance=cluster1 2023-08-26T13:52:24.215 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ shift 2023-08-26T13:52:24.215 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local cluster=cluster1 2023-08-26T13:52:24.215 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local instance=cluster1 2023-08-26T13:52:24.216 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local loop_instance 2023-08-26T13:52:24.216 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-08-26T13:52:24.216 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ sleep 0 2023-08-26T13:52:24.216 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ '[' cluster1 '!=' cluster1 ']' 2023-08-26T13:52:24.216 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ seq 0 3 2023-08-26T13:52:24.219 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-08-26T13:52:24.219 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_5 2023-08-26T13:52:24.222 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local cluster=cluster1:0 2023-08-26T13:52:24.222 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ shift 2023-08-26T13:52:24.223 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local instance 2023-08-26T13:52:24.223 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ set_cluster_instance cluster1:0 cluster instance 2023-08-26T13:52:24.223 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local val=cluster1:0 2023-08-26T13:52:24.223 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local cluster_var_name=cluster 2023-08-26T13:52:24.223 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local instance_var_name=instance 2023-08-26T13:52:24.224 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ cluster=cluster1 2023-08-26T13:52:24.224 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ instance=0 2023-08-26T13:52:24.224 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ '[' 0 = cluster1:0 ']' 2023-08-26T13:52:24.224 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ eval cluster=cluster1 2023-08-26T13:52:24.224 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ cluster=cluster1 2023-08-26T13:52:24.224 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ eval instance=0 2023-08-26T13:52:24.225 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ instance=0 2023-08-26T13:52:24.225 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ daemon_asok_file cluster1:0 cluster1 2023-08-26T13:52:24.225 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ local local_cluster=cluster1:0 2023-08-26T13:52:24.225 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ local cluster=cluster1 2023-08-26T13:52:24.225 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ local instance 2023-08-26T13:52:24.226 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2023-08-26T13:52:24.226 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ local val=cluster1:0 2023-08-26T13:52:24.226 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ local cluster_var_name=local_cluster 2023-08-26T13:52:24.226 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ local instance_var_name=instance 2023-08-26T13:52:24.226 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ cluster=cluster1 2023-08-26T13:52:24.226 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ instance=0 2023-08-26T13:52:24.227 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ '[' 0 = cluster1:0 ']' 2023-08-26T13:52:24.227 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ eval local_cluster=cluster1 2023-08-26T13:52:24.227 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++++ local_cluster=cluster1 2023-08-26T13:52:24.227 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ eval instance=0 2023-08-26T13:52:24.227 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++++ instance=0 2023-08-26T13:52:24.228 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2023-08-26T13:52:24.240 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-08-26T13:52:24.240 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-08-26T13:52:24.240 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-08-26T13:52:24.240 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_5 2023-08-26T13:52:24.349 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:Invalid command: unused arguments: ['mirror/image_5'] 2023-08-26T13:52:24.350 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:rbd mirror status : get status for rbd mirror 2023-08-26T13:52:24.350 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:admin_socket: invalid command 2023-08-26T13:52:24.353 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-08-26T13:52:24.353 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_5 2023-08-26T13:52:24.353 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local cluster=cluster1:1 2023-08-26T13:52:24.353 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ shift 2023-08-26T13:52:24.354 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local instance 2023-08-26T13:52:24.354 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ set_cluster_instance cluster1:1 cluster instance 2023-08-26T13:52:24.354 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local val=cluster1:1 2023-08-26T13:52:24.354 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local cluster_var_name=cluster 2023-08-26T13:52:24.354 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local instance_var_name=instance 2023-08-26T13:52:24.355 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ cluster=cluster1 2023-08-26T13:52:24.355 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ instance=1 2023-08-26T13:52:24.355 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ '[' 1 = cluster1:1 ']' 2023-08-26T13:52:24.355 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ eval cluster=cluster1 2023-08-26T13:52:24.355 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ cluster=cluster1 2023-08-26T13:52:24.356 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ eval instance=1 2023-08-26T13:52:24.356 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ instance=1 2023-08-26T13:52:24.356 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ daemon_asok_file cluster1:1 cluster1 2023-08-26T13:52:24.356 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ local local_cluster=cluster1:1 2023-08-26T13:52:24.356 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ local cluster=cluster1 2023-08-26T13:52:24.357 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ local instance 2023-08-26T13:52:24.357 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2023-08-26T13:52:24.357 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ local val=cluster1:1 2023-08-26T13:52:24.357 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ local cluster_var_name=local_cluster 2023-08-26T13:52:24.357 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ local instance_var_name=instance 2023-08-26T13:52:24.358 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ cluster=cluster1 2023-08-26T13:52:24.358 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ instance=1 2023-08-26T13:52:24.358 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ '[' 1 = cluster1:1 ']' 2023-08-26T13:52:24.358 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ eval local_cluster=cluster1 2023-08-26T13:52:24.358 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++++ local_cluster=cluster1 2023-08-26T13:52:24.358 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ eval instance=1 2023-08-26T13:52:24.359 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++++ instance=1 2023-08-26T13:52:24.359 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2023-08-26T13:52:24.371 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-08-26T13:52:24.372 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-08-26T13:52:24.372 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-08-26T13:52:24.372 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_5 2023-08-26T13:52:24.477 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:Invalid command: unused arguments: ['mirror/image_5'] 2023-08-26T13:52:24.478 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:rbd mirror status : get status for rbd mirror 2023-08-26T13:52:24.478 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:admin_socket: invalid command 2023-08-26T13:52:24.482 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-08-26T13:52:24.482 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/image_5 2023-08-26T13:52:24.482 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local cluster=cluster1:2 2023-08-26T13:52:24.482 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ shift 2023-08-26T13:52:24.483 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local instance 2023-08-26T13:52:24.483 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ set_cluster_instance cluster1:2 cluster instance 2023-08-26T13:52:24.483 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local val=cluster1:2 2023-08-26T13:52:24.483 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local cluster_var_name=cluster 2023-08-26T13:52:24.483 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local instance_var_name=instance 2023-08-26T13:52:24.484 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ cluster=cluster1 2023-08-26T13:52:24.484 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ instance=2 2023-08-26T13:52:24.484 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ '[' 2 = cluster1:2 ']' 2023-08-26T13:52:24.484 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ eval cluster=cluster1 2023-08-26T13:52:24.485 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ cluster=cluster1 2023-08-26T13:52:24.485 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ eval instance=2 2023-08-26T13:52:24.485 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ instance=2 2023-08-26T13:52:24.485 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ daemon_asok_file cluster1:2 cluster1 2023-08-26T13:52:24.485 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ local local_cluster=cluster1:2 2023-08-26T13:52:24.486 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ local cluster=cluster1 2023-08-26T13:52:24.486 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ local instance 2023-08-26T13:52:24.486 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2023-08-26T13:52:24.486 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ local val=cluster1:2 2023-08-26T13:52:24.487 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ local cluster_var_name=local_cluster 2023-08-26T13:52:24.487 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ local instance_var_name=instance 2023-08-26T13:52:24.487 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ cluster=cluster1 2023-08-26T13:52:24.487 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ instance=2 2023-08-26T13:52:24.488 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ '[' 2 = cluster1:2 ']' 2023-08-26T13:52:24.488 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ eval local_cluster=cluster1 2023-08-26T13:52:24.488 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++++ local_cluster=cluster1 2023-08-26T13:52:24.488 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ eval instance=2 2023-08-26T13:52:24.488 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++++ instance=2 2023-08-26T13:52:24.489 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2023-08-26T13:52:24.499 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-08-26T13:52:24.500 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-08-26T13:52:24.500 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-08-26T13:52:24.500 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/image_5 2023-08-26T13:52:24.604 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:no valid command found; 1 closest matches: 2023-08-26T13:52:24.604 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:rbd mirror status mirror/image_50 2023-08-26T13:52:24.605 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:admin_socket: invalid command 2023-08-26T13:52:24.608 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-08-26T13:52:24.608 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ admin_daemon cluster1:3 rbd mirror status mirror/image_5 2023-08-26T13:52:24.608 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local cluster=cluster1:3 2023-08-26T13:52:24.608 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ shift 2023-08-26T13:52:24.608 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local instance 2023-08-26T13:52:24.609 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ set_cluster_instance cluster1:3 cluster instance 2023-08-26T13:52:24.609 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local val=cluster1:3 2023-08-26T13:52:24.609 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local cluster_var_name=cluster 2023-08-26T13:52:24.609 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local instance_var_name=instance 2023-08-26T13:52:24.609 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ cluster=cluster1 2023-08-26T13:52:24.609 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ instance=3 2023-08-26T13:52:24.610 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ '[' 3 = cluster1:3 ']' 2023-08-26T13:52:24.610 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ eval cluster=cluster1 2023-08-26T13:52:24.610 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ cluster=cluster1 2023-08-26T13:52:24.610 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ eval instance=3 2023-08-26T13:52:24.610 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ instance=3 2023-08-26T13:52:24.611 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ daemon_asok_file cluster1:3 cluster1 2023-08-26T13:52:24.611 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ local local_cluster=cluster1:3 2023-08-26T13:52:24.611 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ local cluster=cluster1 2023-08-26T13:52:24.611 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ local instance 2023-08-26T13:52:24.611 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ set_cluster_instance cluster1:3 local_cluster instance 2023-08-26T13:52:24.611 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ local val=cluster1:3 2023-08-26T13:52:24.612 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ local cluster_var_name=local_cluster 2023-08-26T13:52:24.612 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ local instance_var_name=instance 2023-08-26T13:52:24.612 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ cluster=cluster1 2023-08-26T13:52:24.612 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ instance=3 2023-08-26T13:52:24.613 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ '[' 3 = cluster1:3 ']' 2023-08-26T13:52:24.613 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ eval local_cluster=cluster1 2023-08-26T13:52:24.613 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++++ local_cluster=cluster1 2023-08-26T13:52:24.614 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ eval instance=3 2023-08-26T13:52:24.614 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++++ instance=3 2023-08-26T13:52:24.615 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2023-08-26T13:52:24.627 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2023-08-26T13:52:24.627 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2023-08-26T13:52:24.627 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2023-08-26T13:52:24.627 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror status mirror/image_5 2023-08-26T13:52:24.742 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ return 0 2023-08-26T13:52:24.743 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ status_result=' "state": "Replaying"' 2023-08-26T13:52:24.743 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ echo ' "state": "Replaying"' 2023-08-26T13:52:24.743 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ grep -i Replaying 2023-08-26T13:52:24.744 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: "state": "Replaying" 2023-08-26T13:52:24.746 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ current_state=started 2023-08-26T13:52:24.746 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ test started = started 2023-08-26T13:52:24.746 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ return 0 2023-08-26T13:52:24.746 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_5 2023-08-26T13:52:24.746 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local local_cluster=cluster1 2023-08-26T13:52:24.747 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster2 2023-08-26T13:52:24.747 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T13:52:24.747 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=image_5 2023-08-26T13:52:24.747 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ '[' snapshot = journal ']' 2023-08-26T13:52:24.748 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ '[' snapshot = snapshot ']' 2023-08-26T13:52:24.748 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_5 2023-08-26T13:52:24.748 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local local_cluster=cluster1 2023-08-26T13:52:24.749 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster2 2023-08-26T13:52:24.749 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T13:52:24.749 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=image_5 2023-08-26T13:52:24.749 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ mkfname cluster2-mirror-image_5.status 2023-08-26T13:52:24.749 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ echo cluster2-mirror-image_5.status 2023-08-26T13:52:24.750 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ sed -e 's|[/ ]|_|g' 2023-08-26T13:52:24.750 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local status_log=/tmp/tmp.TfDn6BcQ3N/cluster2-mirror-image_5.status 2023-08-26T13:52:24.750 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ mkfname cluster1-mirror-image_5.status 2023-08-26T13:52:24.750 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ echo cluster1-mirror-image_5.status 2023-08-26T13:52:24.750 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ sed -e 's|[/ ]|_|g' 2023-08-26T13:52:24.751 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local local_status_log=/tmp/tmp.TfDn6BcQ3N/cluster1-mirror-image_5.status 2023-08-26T13:52:24.751 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ mirror_image_snapshot cluster2 mirror image_5 2023-08-26T13:52:24.752 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster2 2023-08-26T13:52:24.752 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T13:52:24.752 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=image_5 2023-08-26T13:52:24.752 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_5 2023-08-26T13:52:25.217 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:Snapshot ID: 127 2023-08-26T13:52:25.226 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_5 /tmp/tmp.TfDn6BcQ3N/cluster2-mirror-image_5.status 2023-08-26T13:52:25.226 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster2 2023-08-26T13:52:25.226 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T13:52:25.227 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=image_5 2023-08-26T13:52:25.227 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local log=/tmp/tmp.TfDn6BcQ3N/cluster2-mirror-image_5.status 2023-08-26T13:52:25.227 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_5 --format xml 2023-08-26T13:52:25.227 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-08-26T13:52:25.269 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ xmlstarlet sel -t -v //snapshot/id 2023-08-26T13:52:25.271 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local snapshot_id=127 2023-08-26T13:52:25.271 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ true 2023-08-26T13:52:25.271 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-08-26T13:52:25.271 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ sleep 0.2 2023-08-26T13:52:25.472 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_5 /tmp/tmp.TfDn6BcQ3N/cluster1-mirror-image_5.status 2023-08-26T13:52:25.473 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster1 2023-08-26T13:52:25.473 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T13:52:25.473 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=image_5 2023-08-26T13:52:25.473 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local log=/tmp/tmp.TfDn6BcQ3N/cluster1-mirror-image_5.status 2023-08-26T13:52:25.473 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_5 --format xml 2023-08-26T13:52:25.474 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-08-26T13:52:25.526 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-08-26T13:52:25.528 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local primary_snapshot_id=72 2023-08-26T13:52:25.529 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ test 127 = 72 2023-08-26T13:52:25.529 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-08-26T13:52:25.529 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ sleep 0.4 2023-08-26T13:52:25.929 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_5 /tmp/tmp.TfDn6BcQ3N/cluster1-mirror-image_5.status 2023-08-26T13:52:25.929 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster1 2023-08-26T13:52:25.930 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T13:52:25.930 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=image_5 2023-08-26T13:52:25.930 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local log=/tmp/tmp.TfDn6BcQ3N/cluster1-mirror-image_5.status 2023-08-26T13:52:25.930 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_5 --format xml 2023-08-26T13:52:25.930 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-08-26T13:52:26.053 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-08-26T13:52:26.055 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local primary_snapshot_id=72 2023-08-26T13:52:26.055 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ test 127 = 72 2023-08-26T13:52:26.056 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-08-26T13:52:26.056 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ sleep 0.8 2023-08-26T13:52:26.856 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_5 /tmp/tmp.TfDn6BcQ3N/cluster1-mirror-image_5.status 2023-08-26T13:52:26.857 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster1 2023-08-26T13:52:26.857 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T13:52:26.857 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=image_5 2023-08-26T13:52:26.857 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local log=/tmp/tmp.TfDn6BcQ3N/cluster1-mirror-image_5.status 2023-08-26T13:52:26.858 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_5 --format xml 2023-08-26T13:52:26.858 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-08-26T13:52:26.907 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ true 2023-08-26T13:52:26.907 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-08-26T13:52:26.908 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:-:1.1: Document is empty 2023-08-26T13:52:26.908 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr: 2023-08-26T13:52:26.909 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:^ 2023-08-26T13:52:26.910 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local primary_snapshot_id= 2023-08-26T13:52:26.910 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ test 127 = '' 2023-08-26T13:52:26.910 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-08-26T13:52:26.910 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ sleep 1.6 2023-08-26T13:52:28.510 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_5 /tmp/tmp.TfDn6BcQ3N/cluster1-mirror-image_5.status 2023-08-26T13:52:28.511 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster1 2023-08-26T13:52:28.511 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T13:52:28.511 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=image_5 2023-08-26T13:52:28.511 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local log=/tmp/tmp.TfDn6BcQ3N/cluster1-mirror-image_5.status 2023-08-26T13:52:28.512 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_5 --format xml 2023-08-26T13:52:28.512 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-08-26T13:52:28.561 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-08-26T13:52:28.564 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local primary_snapshot_id=127 2023-08-26T13:52:28.564 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ test 127 = 127 2023-08-26T13:52:28.564 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ return 0 2023-08-26T13:52:28.565 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ wait_for_snap_present cluster1 mirror image_5 snap 2023-08-26T13:52:28.565 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster1 2023-08-26T13:52:28.565 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T13:52:28.565 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=image_5 2023-08-26T13:52:28.565 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local snap_name=snap 2023-08-26T13:52:28.566 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local s 2023-08-26T13:52:28.566 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2023-08-26T13:52:28.566 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ sleep 1 2023-08-26T13:52:29.565 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rbd --cluster cluster1 info mirror/image_5@snap 2023-08-26T13:52:29.611 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:rbd image 'image_5': 2023-08-26T13:52:29.612 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: size 128 MiB in 32 objects 2023-08-26T13:52:29.612 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: order 22 (4 MiB objects) 2023-08-26T13:52:29.612 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: snapshot_count: 2 2023-08-26T13:52:29.612 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: id: 10b7986d5575 2023-08-26T13:52:29.612 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: block_name_prefix: rbd_data.10b7986d5575 2023-08-26T13:52:29.613 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: format: 2 2023-08-26T13:52:29.613 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: features: layering, exclusive-lock, non-primary 2023-08-26T13:52:29.613 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: op_features: 2023-08-26T13:52:29.613 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: flags: 2023-08-26T13:52:29.613 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: create_timestamp: Sat Aug 26 13:43:30 2023 2023-08-26T13:52:29.613 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: access_timestamp: Sat Aug 26 13:43:30 2023 2023-08-26T13:52:29.614 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: modify_timestamp: Sat Aug 26 13:43:30 2023 2023-08-26T13:52:29.614 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: protected: False 2023-08-26T13:52:29.614 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: mirroring state: enabled 2023-08-26T13:52:29.614 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: mirroring mode: snapshot 2023-08-26T13:52:29.614 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: mirroring global id: fd599976-dbcc-4e1b-98dd-2440bc289607 2023-08-26T13:52:29.615 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: mirroring primary: false 2023-08-26T13:52:29.616 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ return 0 2023-08-26T13:52:29.616 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ compare_image_snaps mirror image_5 snap 2023-08-26T13:52:29.616 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T13:52:29.617 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=image_5 2023-08-26T13:52:29.617 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local snap_name=snap 2023-08-26T13:52:29.617 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local ret=0 2023-08-26T13:52:29.617 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local rmt_export=/tmp/tmp.TfDn6BcQ3N/cluster2-mirror-image_5.export 2023-08-26T13:52:29.617 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local loc_export=/tmp/tmp.TfDn6BcQ3N/cluster1-mirror-image_5.export 2023-08-26T13:52:29.618 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rm -f /tmp/tmp.TfDn6BcQ3N/cluster2-mirror-image_5.export /tmp/tmp.TfDn6BcQ3N/cluster1-mirror-image_5.export 2023-08-26T13:52:29.618 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rbd --cluster cluster2 -p mirror export image_5@snap /tmp/tmp.TfDn6BcQ3N/cluster2-mirror-image_5.export 2023-08-26T13:52:29.803 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-08-26T13:52:29.809 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rbd --cluster cluster1 -p mirror export image_5@snap /tmp/tmp.TfDn6BcQ3N/cluster1-mirror-image_5.export 2023-08-26T13:52:30.019 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-08-26T13:52:30.025 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ cmp /tmp/tmp.TfDn6BcQ3N/cluster2-mirror-image_5.export /tmp/tmp.TfDn6BcQ3N/cluster1-mirror-image_5.export 2023-08-26T13:52:30.080 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rm -f /tmp/tmp.TfDn6BcQ3N/cluster2-mirror-image_5.export /tmp/tmp.TfDn6BcQ3N/cluster1-mirror-image_5.export 2023-08-26T13:52:30.117 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ return 0 2023-08-26T13:52:30.117 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-08-26T13:52:30.117 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ image=image_6 2023-08-26T13:52:30.118 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ create_snap cluster2 mirror image_6 snap 2023-08-26T13:52:30.118 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster2 2023-08-26T13:52:30.118 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T13:52:30.118 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=image_6 2023-08-26T13:52:30.118 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local snap_name=snap 2023-08-26T13:52:30.119 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rbd --cluster cluster2 -p mirror snap create image_6@snap --debug-rbd=20 --debug-journaler=20 2023-08-26T13:52:31.237 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ wait_for_image_replay_started cluster1 mirror image_6 2023-08-26T13:52:31.237 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster1 2023-08-26T13:52:31.237 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T13:52:31.237 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=image_6 2023-08-26T13:52:31.238 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ wait_for_image_replay_state cluster1 mirror image_6 started 2023-08-26T13:52:31.238 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster1 2023-08-26T13:52:31.238 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T13:52:31.238 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=image_6 2023-08-26T13:52:31.238 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local state=started 2023-08-26T13:52:31.239 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local s 2023-08-26T13:52:31.239 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-08-26T13:52:31.239 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ sleep 1 2023-08-26T13:52:32.238 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ test_image_replay_state cluster1 mirror image_6 started 2023-08-26T13:52:32.238 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster1 2023-08-26T13:52:32.239 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T13:52:32.239 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=image_6 2023-08-26T13:52:32.239 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local test_state=started 2023-08-26T13:52:32.239 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local status_result 2023-08-26T13:52:32.239 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local current_state=stopped 2023-08-26T13:52:32.240 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_6 2023-08-26T13:52:32.241 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local cluster_instance=cluster1 2023-08-26T13:52:32.241 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ shift 2023-08-26T13:52:32.241 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local cluster=cluster1 2023-08-26T13:52:32.241 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ grep -i state 2023-08-26T13:52:32.241 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local instance=cluster1 2023-08-26T13:52:32.242 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local loop_instance 2023-08-26T13:52:32.242 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-08-26T13:52:32.242 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ sleep 0 2023-08-26T13:52:32.242 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ '[' cluster1 '!=' cluster1 ']' 2023-08-26T13:52:32.242 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ seq 0 3 2023-08-26T13:52:32.244 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-08-26T13:52:32.244 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_6 2023-08-26T13:52:32.245 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local cluster=cluster1:0 2023-08-26T13:52:32.245 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ shift 2023-08-26T13:52:32.245 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local instance 2023-08-26T13:52:32.245 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ set_cluster_instance cluster1:0 cluster instance 2023-08-26T13:52:32.245 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local val=cluster1:0 2023-08-26T13:52:32.245 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local cluster_var_name=cluster 2023-08-26T13:52:32.246 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local instance_var_name=instance 2023-08-26T13:52:32.246 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ cluster=cluster1 2023-08-26T13:52:32.246 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ instance=0 2023-08-26T13:52:32.246 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ '[' 0 = cluster1:0 ']' 2023-08-26T13:52:32.246 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ eval cluster=cluster1 2023-08-26T13:52:32.247 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ cluster=cluster1 2023-08-26T13:52:32.247 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ eval instance=0 2023-08-26T13:52:32.247 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ instance=0 2023-08-26T13:52:32.247 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ daemon_asok_file cluster1:0 cluster1 2023-08-26T13:52:32.249 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ local local_cluster=cluster1:0 2023-08-26T13:52:32.249 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ local cluster=cluster1 2023-08-26T13:52:32.249 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ local instance 2023-08-26T13:52:32.250 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2023-08-26T13:52:32.250 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ local val=cluster1:0 2023-08-26T13:52:32.250 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ local cluster_var_name=local_cluster 2023-08-26T13:52:32.250 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ local instance_var_name=instance 2023-08-26T13:52:32.250 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ cluster=cluster1 2023-08-26T13:52:32.251 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ instance=0 2023-08-26T13:52:32.251 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ '[' 0 = cluster1:0 ']' 2023-08-26T13:52:32.251 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ eval local_cluster=cluster1 2023-08-26T13:52:32.251 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++++ local_cluster=cluster1 2023-08-26T13:52:32.251 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ eval instance=0 2023-08-26T13:52:32.252 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++++ instance=0 2023-08-26T13:52:32.252 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2023-08-26T13:52:32.265 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-08-26T13:52:32.265 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-08-26T13:52:32.265 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-08-26T13:52:32.265 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_6 2023-08-26T13:52:32.376 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:Invalid command: unused arguments: ['mirror/image_6'] 2023-08-26T13:52:32.376 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:rbd mirror status : get status for rbd mirror 2023-08-26T13:52:32.376 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:admin_socket: invalid command 2023-08-26T13:52:32.379 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-08-26T13:52:32.380 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_6 2023-08-26T13:52:32.380 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local cluster=cluster1:1 2023-08-26T13:52:32.380 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ shift 2023-08-26T13:52:32.380 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local instance 2023-08-26T13:52:32.380 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ set_cluster_instance cluster1:1 cluster instance 2023-08-26T13:52:32.381 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local val=cluster1:1 2023-08-26T13:52:32.381 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local cluster_var_name=cluster 2023-08-26T13:52:32.381 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local instance_var_name=instance 2023-08-26T13:52:32.381 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ cluster=cluster1 2023-08-26T13:52:32.381 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ instance=1 2023-08-26T13:52:32.382 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ '[' 1 = cluster1:1 ']' 2023-08-26T13:52:32.382 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ eval cluster=cluster1 2023-08-26T13:52:32.382 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ cluster=cluster1 2023-08-26T13:52:32.382 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ eval instance=1 2023-08-26T13:52:32.383 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ instance=1 2023-08-26T13:52:32.383 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ daemon_asok_file cluster1:1 cluster1 2023-08-26T13:52:32.383 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ local local_cluster=cluster1:1 2023-08-26T13:52:32.383 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ local cluster=cluster1 2023-08-26T13:52:32.384 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ local instance 2023-08-26T13:52:32.384 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2023-08-26T13:52:32.384 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ local val=cluster1:1 2023-08-26T13:52:32.384 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ local cluster_var_name=local_cluster 2023-08-26T13:52:32.385 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ local instance_var_name=instance 2023-08-26T13:52:32.385 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ cluster=cluster1 2023-08-26T13:52:32.385 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ instance=1 2023-08-26T13:52:32.385 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ '[' 1 = cluster1:1 ']' 2023-08-26T13:52:32.386 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ eval local_cluster=cluster1 2023-08-26T13:52:32.386 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++++ local_cluster=cluster1 2023-08-26T13:52:32.386 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ eval instance=1 2023-08-26T13:52:32.387 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++++ instance=1 2023-08-26T13:52:32.387 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2023-08-26T13:52:32.398 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-08-26T13:52:32.398 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-08-26T13:52:32.398 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-08-26T13:52:32.399 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_6 2023-08-26T13:52:32.512 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ return 0 2023-08-26T13:52:32.513 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ status_result=' "state": "Replaying"' 2023-08-26T13:52:32.513 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ echo ' "state": "Replaying"' 2023-08-26T13:52:32.513 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ grep -i Replaying 2023-08-26T13:52:32.514 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: "state": "Replaying" 2023-08-26T13:52:32.516 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ current_state=started 2023-08-26T13:52:32.516 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ test started = started 2023-08-26T13:52:32.516 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ return 0 2023-08-26T13:52:32.516 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_6 2023-08-26T13:52:32.517 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local local_cluster=cluster1 2023-08-26T13:52:32.517 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster2 2023-08-26T13:52:32.517 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T13:52:32.517 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=image_6 2023-08-26T13:52:32.517 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ '[' snapshot = journal ']' 2023-08-26T13:52:32.517 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ '[' snapshot = snapshot ']' 2023-08-26T13:52:32.518 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_6 2023-08-26T13:52:32.518 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local local_cluster=cluster1 2023-08-26T13:52:32.518 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster2 2023-08-26T13:52:32.518 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T13:52:32.518 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=image_6 2023-08-26T13:52:32.519 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ mkfname cluster2-mirror-image_6.status 2023-08-26T13:52:32.519 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ echo cluster2-mirror-image_6.status 2023-08-26T13:52:32.519 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ sed -e 's|[/ ]|_|g' 2023-08-26T13:52:32.519 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local status_log=/tmp/tmp.TfDn6BcQ3N/cluster2-mirror-image_6.status 2023-08-26T13:52:32.519 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ mkfname cluster1-mirror-image_6.status 2023-08-26T13:52:32.520 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ echo cluster1-mirror-image_6.status 2023-08-26T13:52:32.520 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ sed -e 's|[/ ]|_|g' 2023-08-26T13:52:32.521 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local local_status_log=/tmp/tmp.TfDn6BcQ3N/cluster1-mirror-image_6.status 2023-08-26T13:52:32.521 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ mirror_image_snapshot cluster2 mirror image_6 2023-08-26T13:52:32.521 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster2 2023-08-26T13:52:32.521 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T13:52:32.521 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=image_6 2023-08-26T13:52:32.522 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_6 2023-08-26T13:52:33.245 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:Snapshot ID: 129 2023-08-26T13:52:33.254 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_6 /tmp/tmp.TfDn6BcQ3N/cluster2-mirror-image_6.status 2023-08-26T13:52:33.255 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster2 2023-08-26T13:52:33.255 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T13:52:33.255 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=image_6 2023-08-26T13:52:33.255 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local log=/tmp/tmp.TfDn6BcQ3N/cluster2-mirror-image_6.status 2023-08-26T13:52:33.255 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_6 --format xml 2023-08-26T13:52:33.256 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-08-26T13:52:33.298 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ xmlstarlet sel -t -v //snapshot/id 2023-08-26T13:52:33.299 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local snapshot_id=129 2023-08-26T13:52:33.300 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ true 2023-08-26T13:52:33.300 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-08-26T13:52:33.300 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ sleep 0.2 2023-08-26T13:52:33.501 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_6 /tmp/tmp.TfDn6BcQ3N/cluster1-mirror-image_6.status 2023-08-26T13:52:33.501 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster1 2023-08-26T13:52:33.501 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T13:52:33.502 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=image_6 2023-08-26T13:52:33.502 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local log=/tmp/tmp.TfDn6BcQ3N/cluster1-mirror-image_6.status 2023-08-26T13:52:33.502 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_6 --format xml 2023-08-26T13:52:33.502 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-08-26T13:52:33.546 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ true 2023-08-26T13:52:33.547 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-08-26T13:52:33.548 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:-:1.1: Document is empty 2023-08-26T13:52:33.548 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr: 2023-08-26T13:52:33.548 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:^ 2023-08-26T13:52:33.549 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local primary_snapshot_id= 2023-08-26T13:52:33.549 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ test 129 = '' 2023-08-26T13:52:33.550 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-08-26T13:52:33.550 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ sleep 0.4 2023-08-26T13:52:33.950 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_6 /tmp/tmp.TfDn6BcQ3N/cluster1-mirror-image_6.status 2023-08-26T13:52:33.950 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster1 2023-08-26T13:52:33.950 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T13:52:33.950 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=image_6 2023-08-26T13:52:33.951 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local log=/tmp/tmp.TfDn6BcQ3N/cluster1-mirror-image_6.status 2023-08-26T13:52:33.951 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_6 --format xml 2023-08-26T13:52:33.951 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-08-26T13:52:33.998 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ true 2023-08-26T13:52:33.998 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-08-26T13:52:34.000 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:-:1.1: Document is empty 2023-08-26T13:52:34.000 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr: 2023-08-26T13:52:34.000 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:^ 2023-08-26T13:52:34.000 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local primary_snapshot_id= 2023-08-26T13:52:34.000 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ test 129 = '' 2023-08-26T13:52:34.001 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-08-26T13:52:34.001 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ sleep 0.8 2023-08-26T13:52:34.801 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_6 /tmp/tmp.TfDn6BcQ3N/cluster1-mirror-image_6.status 2023-08-26T13:52:34.801 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster1 2023-08-26T13:52:34.802 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T13:52:34.802 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=image_6 2023-08-26T13:52:34.802 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local log=/tmp/tmp.TfDn6BcQ3N/cluster1-mirror-image_6.status 2023-08-26T13:52:34.802 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_6 --format xml 2023-08-26T13:52:34.803 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-08-26T13:52:34.847 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-08-26T13:52:34.850 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local primary_snapshot_id=129 2023-08-26T13:52:34.850 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ test 129 = 129 2023-08-26T13:52:34.850 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ return 0 2023-08-26T13:52:34.850 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ wait_for_snap_present cluster1 mirror image_6 snap 2023-08-26T13:52:34.851 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster1 2023-08-26T13:52:34.851 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T13:52:34.851 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=image_6 2023-08-26T13:52:34.851 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local snap_name=snap 2023-08-26T13:52:34.851 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local s 2023-08-26T13:52:34.852 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2023-08-26T13:52:34.852 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ sleep 1 2023-08-26T13:52:35.850 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rbd --cluster cluster1 info mirror/image_6@snap 2023-08-26T13:52:35.894 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:rbd image 'image_6': 2023-08-26T13:52:35.894 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: size 128 MiB in 32 objects 2023-08-26T13:52:35.894 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: order 22 (4 MiB objects) 2023-08-26T13:52:35.895 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: snapshot_count: 2 2023-08-26T13:52:35.895 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: id: 11d2fde1e1f1 2023-08-26T13:52:35.895 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: block_name_prefix: rbd_data.11d2fde1e1f1 2023-08-26T13:52:35.895 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: format: 2 2023-08-26T13:52:35.896 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: features: layering, exclusive-lock, non-primary 2023-08-26T13:52:35.896 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: op_features: 2023-08-26T13:52:35.896 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: flags: 2023-08-26T13:52:35.896 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: create_timestamp: Sat Aug 26 13:43:37 2023 2023-08-26T13:52:35.897 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: access_timestamp: Sat Aug 26 13:43:37 2023 2023-08-26T13:52:35.897 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: modify_timestamp: Sat Aug 26 13:43:37 2023 2023-08-26T13:52:35.897 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: protected: False 2023-08-26T13:52:35.897 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: mirroring state: enabled 2023-08-26T13:52:35.897 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: mirroring mode: snapshot 2023-08-26T13:52:35.898 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: mirroring global id: 42815663-2c57-4ae9-8fb7-0f989c1a91e4 2023-08-26T13:52:35.898 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: mirroring primary: false 2023-08-26T13:52:35.899 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ return 0 2023-08-26T13:52:35.899 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ compare_image_snaps mirror image_6 snap 2023-08-26T13:52:35.900 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T13:52:35.900 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=image_6 2023-08-26T13:52:35.900 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local snap_name=snap 2023-08-26T13:52:35.900 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local ret=0 2023-08-26T13:52:35.900 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local rmt_export=/tmp/tmp.TfDn6BcQ3N/cluster2-mirror-image_6.export 2023-08-26T13:52:35.901 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local loc_export=/tmp/tmp.TfDn6BcQ3N/cluster1-mirror-image_6.export 2023-08-26T13:52:35.901 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rm -f /tmp/tmp.TfDn6BcQ3N/cluster2-mirror-image_6.export /tmp/tmp.TfDn6BcQ3N/cluster1-mirror-image_6.export 2023-08-26T13:52:35.901 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rbd --cluster cluster2 -p mirror export image_6@snap /tmp/tmp.TfDn6BcQ3N/cluster2-mirror-image_6.export 2023-08-26T13:52:36.055 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-08-26T13:52:36.060 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rbd --cluster cluster1 -p mirror export image_6@snap /tmp/tmp.TfDn6BcQ3N/cluster1-mirror-image_6.export 2023-08-26T13:52:36.276 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-08-26T13:52:36.282 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ cmp /tmp/tmp.TfDn6BcQ3N/cluster2-mirror-image_6.export /tmp/tmp.TfDn6BcQ3N/cluster1-mirror-image_6.export 2023-08-26T13:52:36.337 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rm -f /tmp/tmp.TfDn6BcQ3N/cluster2-mirror-image_6.export /tmp/tmp.TfDn6BcQ3N/cluster1-mirror-image_6.export 2023-08-26T13:52:36.375 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ return 0 2023-08-26T13:52:36.376 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-08-26T13:52:36.376 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ image=image_7 2023-08-26T13:52:36.376 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ create_snap cluster2 mirror image_7 snap 2023-08-26T13:52:36.376 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster2 2023-08-26T13:52:36.376 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T13:52:36.377 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=image_7 2023-08-26T13:52:36.377 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local snap_name=snap 2023-08-26T13:52:36.377 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rbd --cluster cluster2 -p mirror snap create image_7@snap --debug-rbd=20 --debug-journaler=20 2023-08-26T13:52:38.268 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ wait_for_image_replay_started cluster1 mirror image_7 2023-08-26T13:52:38.268 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster1 2023-08-26T13:52:38.268 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T13:52:38.269 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=image_7 2023-08-26T13:52:38.269 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ wait_for_image_replay_state cluster1 mirror image_7 started 2023-08-26T13:52:38.269 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster1 2023-08-26T13:52:38.269 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T13:52:38.269 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=image_7 2023-08-26T13:52:38.270 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local state=started 2023-08-26T13:52:38.270 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local s 2023-08-26T13:52:38.270 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-08-26T13:52:38.270 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ sleep 1 2023-08-26T13:52:39.270 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ test_image_replay_state cluster1 mirror image_7 started 2023-08-26T13:52:39.270 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster1 2023-08-26T13:52:39.270 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T13:52:39.271 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=image_7 2023-08-26T13:52:39.271 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local test_state=started 2023-08-26T13:52:39.271 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local status_result 2023-08-26T13:52:39.271 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local current_state=stopped 2023-08-26T13:52:39.271 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_7 2023-08-26T13:52:39.272 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local cluster_instance=cluster1 2023-08-26T13:52:39.272 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ grep -i state 2023-08-26T13:52:39.272 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ shift 2023-08-26T13:52:39.272 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local cluster=cluster1 2023-08-26T13:52:39.272 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local instance=cluster1 2023-08-26T13:52:39.272 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local loop_instance 2023-08-26T13:52:39.273 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-08-26T13:52:39.273 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ sleep 0 2023-08-26T13:52:39.275 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ '[' cluster1 '!=' cluster1 ']' 2023-08-26T13:52:39.275 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ seq 0 3 2023-08-26T13:52:39.275 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-08-26T13:52:39.276 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_7 2023-08-26T13:52:39.276 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local cluster=cluster1:0 2023-08-26T13:52:39.276 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ shift 2023-08-26T13:52:39.276 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local instance 2023-08-26T13:52:39.276 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ set_cluster_instance cluster1:0 cluster instance 2023-08-26T13:52:39.277 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local val=cluster1:0 2023-08-26T13:52:39.277 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local cluster_var_name=cluster 2023-08-26T13:52:39.277 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local instance_var_name=instance 2023-08-26T13:52:39.277 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ cluster=cluster1 2023-08-26T13:52:39.277 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ instance=0 2023-08-26T13:52:39.278 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ '[' 0 = cluster1:0 ']' 2023-08-26T13:52:39.278 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ eval cluster=cluster1 2023-08-26T13:52:39.278 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ cluster=cluster1 2023-08-26T13:52:39.278 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ eval instance=0 2023-08-26T13:52:39.278 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ instance=0 2023-08-26T13:52:39.279 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ daemon_asok_file cluster1:0 cluster1 2023-08-26T13:52:39.279 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ local local_cluster=cluster1:0 2023-08-26T13:52:39.279 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ local cluster=cluster1 2023-08-26T13:52:39.279 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ local instance 2023-08-26T13:52:39.279 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2023-08-26T13:52:39.279 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ local val=cluster1:0 2023-08-26T13:52:39.280 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ local cluster_var_name=local_cluster 2023-08-26T13:52:39.280 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ local instance_var_name=instance 2023-08-26T13:52:39.281 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ cluster=cluster1 2023-08-26T13:52:39.281 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ instance=0 2023-08-26T13:52:39.281 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ '[' 0 = cluster1:0 ']' 2023-08-26T13:52:39.282 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ eval local_cluster=cluster1 2023-08-26T13:52:39.282 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++++ local_cluster=cluster1 2023-08-26T13:52:39.282 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ eval instance=0 2023-08-26T13:52:39.282 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++++ instance=0 2023-08-26T13:52:39.283 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2023-08-26T13:52:39.295 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-08-26T13:52:39.295 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-08-26T13:52:39.295 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-08-26T13:52:39.296 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_7 2023-08-26T13:52:39.404 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:Invalid command: unused arguments: ['mirror/image_7'] 2023-08-26T13:52:39.405 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:rbd mirror status : get status for rbd mirror 2023-08-26T13:52:39.405 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:admin_socket: invalid command 2023-08-26T13:52:39.407 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-08-26T13:52:39.407 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_7 2023-08-26T13:52:39.408 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local cluster=cluster1:1 2023-08-26T13:52:39.408 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ shift 2023-08-26T13:52:39.408 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local instance 2023-08-26T13:52:39.408 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ set_cluster_instance cluster1:1 cluster instance 2023-08-26T13:52:39.409 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local val=cluster1:1 2023-08-26T13:52:39.409 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local cluster_var_name=cluster 2023-08-26T13:52:39.409 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local instance_var_name=instance 2023-08-26T13:52:39.409 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ cluster=cluster1 2023-08-26T13:52:39.409 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ instance=1 2023-08-26T13:52:39.409 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ '[' 1 = cluster1:1 ']' 2023-08-26T13:52:39.410 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ eval cluster=cluster1 2023-08-26T13:52:39.410 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ cluster=cluster1 2023-08-26T13:52:39.410 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ eval instance=1 2023-08-26T13:52:39.410 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ instance=1 2023-08-26T13:52:39.410 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ daemon_asok_file cluster1:1 cluster1 2023-08-26T13:52:39.411 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ local local_cluster=cluster1:1 2023-08-26T13:52:39.411 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ local cluster=cluster1 2023-08-26T13:52:39.411 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ local instance 2023-08-26T13:52:39.411 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2023-08-26T13:52:39.411 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ local val=cluster1:1 2023-08-26T13:52:39.412 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ local cluster_var_name=local_cluster 2023-08-26T13:52:39.412 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ local instance_var_name=instance 2023-08-26T13:52:39.412 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ cluster=cluster1 2023-08-26T13:52:39.412 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ instance=1 2023-08-26T13:52:39.412 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ '[' 1 = cluster1:1 ']' 2023-08-26T13:52:39.412 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ eval local_cluster=cluster1 2023-08-26T13:52:39.413 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++++ local_cluster=cluster1 2023-08-26T13:52:39.413 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ eval instance=1 2023-08-26T13:52:39.413 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++++ instance=1 2023-08-26T13:52:39.413 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2023-08-26T13:52:39.426 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-08-26T13:52:39.426 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-08-26T13:52:39.426 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-08-26T13:52:39.426 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_7 2023-08-26T13:52:39.532 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:Invalid command: unused arguments: ['mirror/image_7'] 2023-08-26T13:52:39.532 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:rbd mirror status : get status for rbd mirror 2023-08-26T13:52:39.532 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:admin_socket: invalid command 2023-08-26T13:52:39.535 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-08-26T13:52:39.535 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/image_7 2023-08-26T13:52:39.535 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local cluster=cluster1:2 2023-08-26T13:52:39.535 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ shift 2023-08-26T13:52:39.536 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local instance 2023-08-26T13:52:39.536 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ set_cluster_instance cluster1:2 cluster instance 2023-08-26T13:52:39.536 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local val=cluster1:2 2023-08-26T13:52:39.536 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local cluster_var_name=cluster 2023-08-26T13:52:39.536 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local instance_var_name=instance 2023-08-26T13:52:39.537 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ cluster=cluster1 2023-08-26T13:52:39.537 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ instance=2 2023-08-26T13:52:39.537 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ '[' 2 = cluster1:2 ']' 2023-08-26T13:52:39.537 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ eval cluster=cluster1 2023-08-26T13:52:39.537 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ cluster=cluster1 2023-08-26T13:52:39.537 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ eval instance=2 2023-08-26T13:52:39.538 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ instance=2 2023-08-26T13:52:39.538 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ daemon_asok_file cluster1:2 cluster1 2023-08-26T13:52:39.538 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ local local_cluster=cluster1:2 2023-08-26T13:52:39.538 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ local cluster=cluster1 2023-08-26T13:52:39.538 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ local instance 2023-08-26T13:52:39.539 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2023-08-26T13:52:39.539 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ local val=cluster1:2 2023-08-26T13:52:39.539 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ local cluster_var_name=local_cluster 2023-08-26T13:52:39.539 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ local instance_var_name=instance 2023-08-26T13:52:39.539 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ cluster=cluster1 2023-08-26T13:52:39.540 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ instance=2 2023-08-26T13:52:39.540 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ '[' 2 = cluster1:2 ']' 2023-08-26T13:52:39.540 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ eval local_cluster=cluster1 2023-08-26T13:52:39.540 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++++ local_cluster=cluster1 2023-08-26T13:52:39.540 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ eval instance=2 2023-08-26T13:52:39.540 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++++ instance=2 2023-08-26T13:52:39.541 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2023-08-26T13:52:39.553 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-08-26T13:52:39.554 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-08-26T13:52:39.554 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-08-26T13:52:39.554 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/image_7 2023-08-26T13:52:39.667 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ return 0 2023-08-26T13:52:39.667 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ status_result=' "state": "Replaying"' 2023-08-26T13:52:39.668 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ echo ' "state": "Replaying"' 2023-08-26T13:52:39.668 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ grep -i Replaying 2023-08-26T13:52:39.669 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: "state": "Replaying" 2023-08-26T13:52:39.671 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ current_state=started 2023-08-26T13:52:39.671 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ test started = started 2023-08-26T13:52:39.671 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ return 0 2023-08-26T13:52:39.671 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_7 2023-08-26T13:52:39.672 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local local_cluster=cluster1 2023-08-26T13:52:39.672 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster2 2023-08-26T13:52:39.672 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T13:52:39.672 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=image_7 2023-08-26T13:52:39.673 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ '[' snapshot = journal ']' 2023-08-26T13:52:39.673 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ '[' snapshot = snapshot ']' 2023-08-26T13:52:39.673 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_7 2023-08-26T13:52:39.673 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local local_cluster=cluster1 2023-08-26T13:52:39.673 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster2 2023-08-26T13:52:39.673 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T13:52:39.674 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=image_7 2023-08-26T13:52:39.674 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ mkfname cluster2-mirror-image_7.status 2023-08-26T13:52:39.674 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ echo cluster2-mirror-image_7.status 2023-08-26T13:52:39.674 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ sed -e 's|[/ ]|_|g' 2023-08-26T13:52:39.674 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local status_log=/tmp/tmp.TfDn6BcQ3N/cluster2-mirror-image_7.status 2023-08-26T13:52:39.675 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ mkfname cluster1-mirror-image_7.status 2023-08-26T13:52:39.675 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ echo cluster1-mirror-image_7.status 2023-08-26T13:52:39.675 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ sed -e 's|[/ ]|_|g' 2023-08-26T13:52:39.676 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local local_status_log=/tmp/tmp.TfDn6BcQ3N/cluster1-mirror-image_7.status 2023-08-26T13:52:39.676 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ mirror_image_snapshot cluster2 mirror image_7 2023-08-26T13:52:39.676 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster2 2023-08-26T13:52:39.677 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T13:52:39.677 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=image_7 2023-08-26T13:52:39.677 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_7 2023-08-26T13:52:40.271 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:Snapshot ID: 131 2023-08-26T13:52:40.280 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_7 /tmp/tmp.TfDn6BcQ3N/cluster2-mirror-image_7.status 2023-08-26T13:52:40.280 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster2 2023-08-26T13:52:40.281 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T13:52:40.281 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=image_7 2023-08-26T13:52:40.281 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local log=/tmp/tmp.TfDn6BcQ3N/cluster2-mirror-image_7.status 2023-08-26T13:52:40.281 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_7 --format xml 2023-08-26T13:52:40.281 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-08-26T13:52:40.324 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ xmlstarlet sel -t -v //snapshot/id 2023-08-26T13:52:40.326 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local snapshot_id=131 2023-08-26T13:52:40.326 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ true 2023-08-26T13:52:40.327 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-08-26T13:52:40.327 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ sleep 0.2 2023-08-26T13:52:40.528 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_7 /tmp/tmp.TfDn6BcQ3N/cluster1-mirror-image_7.status 2023-08-26T13:52:40.528 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster1 2023-08-26T13:52:40.528 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T13:52:40.529 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=image_7 2023-08-26T13:52:40.529 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local log=/tmp/tmp.TfDn6BcQ3N/cluster1-mirror-image_7.status 2023-08-26T13:52:40.529 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_7 --format xml 2023-08-26T13:52:40.529 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-08-26T13:52:40.576 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-08-26T13:52:40.578 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local primary_snapshot_id=74 2023-08-26T13:52:40.578 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ test 131 = 74 2023-08-26T13:52:40.578 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-08-26T13:52:40.579 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ sleep 0.4 2023-08-26T13:52:40.980 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_7 /tmp/tmp.TfDn6BcQ3N/cluster1-mirror-image_7.status 2023-08-26T13:52:40.980 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster1 2023-08-26T13:52:40.980 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T13:52:40.980 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=image_7 2023-08-26T13:52:40.981 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local log=/tmp/tmp.TfDn6BcQ3N/cluster1-mirror-image_7.status 2023-08-26T13:52:40.981 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_7 --format xml 2023-08-26T13:52:40.981 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-08-26T13:52:41.033 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-08-26T13:52:41.034 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local primary_snapshot_id=74 2023-08-26T13:52:41.035 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ test 131 = 74 2023-08-26T13:52:41.035 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-08-26T13:52:41.035 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ sleep 0.8 2023-08-26T13:52:41.836 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_7 /tmp/tmp.TfDn6BcQ3N/cluster1-mirror-image_7.status 2023-08-26T13:52:41.836 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster1 2023-08-26T13:52:41.837 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T13:52:41.837 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=image_7 2023-08-26T13:52:41.837 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local log=/tmp/tmp.TfDn6BcQ3N/cluster1-mirror-image_7.status 2023-08-26T13:52:41.837 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_7 --format xml 2023-08-26T13:52:41.837 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-08-26T13:52:41.886 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ true 2023-08-26T13:52:41.886 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-08-26T13:52:41.887 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:-:1.1: Document is empty 2023-08-26T13:52:41.887 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr: 2023-08-26T13:52:41.888 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:^ 2023-08-26T13:52:41.888 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local primary_snapshot_id= 2023-08-26T13:52:41.889 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ test 131 = '' 2023-08-26T13:52:41.889 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-08-26T13:52:41.889 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ sleep 1.6 2023-08-26T13:52:43.490 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_7 /tmp/tmp.TfDn6BcQ3N/cluster1-mirror-image_7.status 2023-08-26T13:52:43.490 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster1 2023-08-26T13:52:43.490 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T13:52:43.490 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=image_7 2023-08-26T13:52:43.491 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local log=/tmp/tmp.TfDn6BcQ3N/cluster1-mirror-image_7.status 2023-08-26T13:52:43.491 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_7 --format xml 2023-08-26T13:52:43.491 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-08-26T13:52:43.539 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-08-26T13:52:43.541 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local primary_snapshot_id=131 2023-08-26T13:52:43.542 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ test 131 = 131 2023-08-26T13:52:43.542 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ return 0 2023-08-26T13:52:43.542 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ wait_for_snap_present cluster1 mirror image_7 snap 2023-08-26T13:52:43.542 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster1 2023-08-26T13:52:43.543 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T13:52:43.543 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=image_7 2023-08-26T13:52:43.543 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local snap_name=snap 2023-08-26T13:52:43.543 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local s 2023-08-26T13:52:43.543 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2023-08-26T13:52:43.544 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ sleep 1 2023-08-26T13:52:44.543 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rbd --cluster cluster1 info mirror/image_7@snap 2023-08-26T13:52:44.587 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:rbd image 'image_7': 2023-08-26T13:52:44.587 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: size 128 MiB in 32 objects 2023-08-26T13:52:44.588 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: order 22 (4 MiB objects) 2023-08-26T13:52:44.588 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: snapshot_count: 2 2023-08-26T13:52:44.588 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: id: 122b9ca6f283 2023-08-26T13:52:44.588 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: block_name_prefix: rbd_data.122b9ca6f283 2023-08-26T13:52:44.589 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: format: 2 2023-08-26T13:52:44.589 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: features: layering, exclusive-lock, non-primary 2023-08-26T13:52:44.589 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: op_features: 2023-08-26T13:52:44.589 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: flags: 2023-08-26T13:52:44.589 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: create_timestamp: Sat Aug 26 13:43:44 2023 2023-08-26T13:52:44.589 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: access_timestamp: Sat Aug 26 13:43:44 2023 2023-08-26T13:52:44.590 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: modify_timestamp: Sat Aug 26 13:43:44 2023 2023-08-26T13:52:44.590 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: protected: False 2023-08-26T13:52:44.590 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: mirroring state: enabled 2023-08-26T13:52:44.590 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: mirroring mode: snapshot 2023-08-26T13:52:44.590 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: mirroring global id: cdbf4c2c-a074-4ef2-b5df-d98b1449c167 2023-08-26T13:52:44.591 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: mirroring primary: false 2023-08-26T13:52:44.592 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ return 0 2023-08-26T13:52:44.592 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ compare_image_snaps mirror image_7 snap 2023-08-26T13:52:44.592 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T13:52:44.592 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=image_7 2023-08-26T13:52:44.593 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local snap_name=snap 2023-08-26T13:52:44.593 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local ret=0 2023-08-26T13:52:44.593 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local rmt_export=/tmp/tmp.TfDn6BcQ3N/cluster2-mirror-image_7.export 2023-08-26T13:52:44.593 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local loc_export=/tmp/tmp.TfDn6BcQ3N/cluster1-mirror-image_7.export 2023-08-26T13:52:44.593 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rm -f /tmp/tmp.TfDn6BcQ3N/cluster2-mirror-image_7.export /tmp/tmp.TfDn6BcQ3N/cluster1-mirror-image_7.export 2023-08-26T13:52:44.593 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rbd --cluster cluster2 -p mirror export image_7@snap /tmp/tmp.TfDn6BcQ3N/cluster2-mirror-image_7.export 2023-08-26T13:52:44.766 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-08-26T13:52:44.773 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rbd --cluster cluster1 -p mirror export image_7@snap /tmp/tmp.TfDn6BcQ3N/cluster1-mirror-image_7.export 2023-08-26T13:52:45.032 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-08-26T13:52:45.038 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ cmp /tmp/tmp.TfDn6BcQ3N/cluster2-mirror-image_7.export /tmp/tmp.TfDn6BcQ3N/cluster1-mirror-image_7.export 2023-08-26T13:52:45.094 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rm -f /tmp/tmp.TfDn6BcQ3N/cluster2-mirror-image_7.export /tmp/tmp.TfDn6BcQ3N/cluster1-mirror-image_7.export 2023-08-26T13:52:45.131 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ return 0 2023-08-26T13:52:45.131 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-08-26T13:52:45.132 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ image=image_8 2023-08-26T13:52:45.132 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ create_snap cluster2 mirror image_8 snap 2023-08-26T13:52:45.132 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster2 2023-08-26T13:52:45.132 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T13:52:45.132 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=image_8 2023-08-26T13:52:45.133 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local snap_name=snap 2023-08-26T13:52:45.133 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rbd --cluster cluster2 -p mirror snap create image_8@snap --debug-rbd=20 --debug-journaler=20 2023-08-26T13:52:46.292 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ wait_for_image_replay_started cluster1 mirror image_8 2023-08-26T13:52:46.292 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster1 2023-08-26T13:52:46.292 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T13:52:46.292 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=image_8 2023-08-26T13:52:46.293 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ wait_for_image_replay_state cluster1 mirror image_8 started 2023-08-26T13:52:46.293 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster1 2023-08-26T13:52:46.293 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T13:52:46.293 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=image_8 2023-08-26T13:52:46.294 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local state=started 2023-08-26T13:52:46.294 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local s 2023-08-26T13:52:46.294 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-08-26T13:52:46.294 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ sleep 1 2023-08-26T13:52:46.734 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.0 with skip (0.7025008085740523) > weight (0.25) 2023-08-26T13:52:46.734 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.1 with skip (0.6678589184779917) > weight (0.25) 2023-08-26T13:52:46.735 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.2 with skip (0.5383475638449655) > weight (0.25) 2023-08-26T13:52:46.735 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.3 with skip (0.9784895398515446) > weight (0.25) 2023-08-26T13:52:46.735 INFO:tasks.rbd_mirror_thrash:waiting for 76 secs before thrashing 2023-08-26T13:52:47.293 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ test_image_replay_state cluster1 mirror image_8 started 2023-08-26T13:52:47.293 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster1 2023-08-26T13:52:47.293 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T13:52:47.293 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=image_8 2023-08-26T13:52:47.294 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local test_state=started 2023-08-26T13:52:47.294 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local status_result 2023-08-26T13:52:47.294 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local current_state=stopped 2023-08-26T13:52:47.297 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_8 2023-08-26T13:52:47.297 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local cluster_instance=cluster1 2023-08-26T13:52:47.297 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ shift 2023-08-26T13:52:47.298 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ grep -i state 2023-08-26T13:52:47.298 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local cluster=cluster1 2023-08-26T13:52:47.298 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local instance=cluster1 2023-08-26T13:52:47.298 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local loop_instance 2023-08-26T13:52:47.298 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-08-26T13:52:47.298 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ sleep 0 2023-08-26T13:52:47.299 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ '[' cluster1 '!=' cluster1 ']' 2023-08-26T13:52:47.299 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ seq 0 3 2023-08-26T13:52:47.299 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-08-26T13:52:47.299 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_8 2023-08-26T13:52:47.299 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local cluster=cluster1:0 2023-08-26T13:52:47.300 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ shift 2023-08-26T13:52:47.300 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local instance 2023-08-26T13:52:47.300 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ set_cluster_instance cluster1:0 cluster instance 2023-08-26T13:52:47.300 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local val=cluster1:0 2023-08-26T13:52:47.300 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local cluster_var_name=cluster 2023-08-26T13:52:47.301 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local instance_var_name=instance 2023-08-26T13:52:47.301 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ cluster=cluster1 2023-08-26T13:52:47.301 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ instance=0 2023-08-26T13:52:47.301 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ '[' 0 = cluster1:0 ']' 2023-08-26T13:52:47.301 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ eval cluster=cluster1 2023-08-26T13:52:47.302 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ cluster=cluster1 2023-08-26T13:52:47.302 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ eval instance=0 2023-08-26T13:52:47.302 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ instance=0 2023-08-26T13:52:47.302 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ daemon_asok_file cluster1:0 cluster1 2023-08-26T13:52:47.302 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ local local_cluster=cluster1:0 2023-08-26T13:52:47.302 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ local cluster=cluster1 2023-08-26T13:52:47.303 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ local instance 2023-08-26T13:52:47.303 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2023-08-26T13:52:47.303 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ local val=cluster1:0 2023-08-26T13:52:47.303 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ local cluster_var_name=local_cluster 2023-08-26T13:52:47.304 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ local instance_var_name=instance 2023-08-26T13:52:47.305 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ cluster=cluster1 2023-08-26T13:52:47.305 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ instance=0 2023-08-26T13:52:47.305 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ '[' 0 = cluster1:0 ']' 2023-08-26T13:52:47.305 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ eval local_cluster=cluster1 2023-08-26T13:52:47.305 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++++ local_cluster=cluster1 2023-08-26T13:52:47.306 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ eval instance=0 2023-08-26T13:52:47.306 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++++ instance=0 2023-08-26T13:52:47.306 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2023-08-26T13:52:47.319 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-08-26T13:52:47.319 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-08-26T13:52:47.320 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-08-26T13:52:47.320 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_8 2023-08-26T13:52:47.438 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ return 0 2023-08-26T13:52:47.438 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ status_result=' "state": "Replaying"' 2023-08-26T13:52:47.439 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ echo ' "state": "Replaying"' 2023-08-26T13:52:47.439 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ grep -i Replaying 2023-08-26T13:52:47.439 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: "state": "Replaying" 2023-08-26T13:52:47.441 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ current_state=started 2023-08-26T13:52:47.442 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ test started = started 2023-08-26T13:52:47.442 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ return 0 2023-08-26T13:52:47.442 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_8 2023-08-26T13:52:47.442 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local local_cluster=cluster1 2023-08-26T13:52:47.442 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster2 2023-08-26T13:52:47.443 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T13:52:47.443 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=image_8 2023-08-26T13:52:47.443 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ '[' snapshot = journal ']' 2023-08-26T13:52:47.443 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ '[' snapshot = snapshot ']' 2023-08-26T13:52:47.443 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_8 2023-08-26T13:52:47.444 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local local_cluster=cluster1 2023-08-26T13:52:47.444 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster2 2023-08-26T13:52:47.444 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T13:52:47.444 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=image_8 2023-08-26T13:52:47.444 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ mkfname cluster2-mirror-image_8.status 2023-08-26T13:52:47.445 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ echo cluster2-mirror-image_8.status 2023-08-26T13:52:47.445 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ sed -e 's|[/ ]|_|g' 2023-08-26T13:52:47.445 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local status_log=/tmp/tmp.TfDn6BcQ3N/cluster2-mirror-image_8.status 2023-08-26T13:52:47.445 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ mkfname cluster1-mirror-image_8.status 2023-08-26T13:52:47.445 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ echo cluster1-mirror-image_8.status 2023-08-26T13:52:47.446 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ sed -e 's|[/ ]|_|g' 2023-08-26T13:52:47.446 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local local_status_log=/tmp/tmp.TfDn6BcQ3N/cluster1-mirror-image_8.status 2023-08-26T13:52:47.447 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ mirror_image_snapshot cluster2 mirror image_8 2023-08-26T13:52:47.447 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster2 2023-08-26T13:52:47.447 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T13:52:47.447 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=image_8 2023-08-26T13:52:47.447 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_8 2023-08-26T13:52:48.289 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:Snapshot ID: 133 2023-08-26T13:52:48.298 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_8 /tmp/tmp.TfDn6BcQ3N/cluster2-mirror-image_8.status 2023-08-26T13:52:48.299 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster2 2023-08-26T13:52:48.299 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T13:52:48.299 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=image_8 2023-08-26T13:52:48.299 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local log=/tmp/tmp.TfDn6BcQ3N/cluster2-mirror-image_8.status 2023-08-26T13:52:48.299 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_8 --format xml 2023-08-26T13:52:48.300 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-08-26T13:52:48.347 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ xmlstarlet sel -t -v //snapshot/id 2023-08-26T13:52:48.348 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local snapshot_id=133 2023-08-26T13:52:48.349 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ true 2023-08-26T13:52:48.349 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-08-26T13:52:48.349 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ sleep 0.2 2023-08-26T13:52:48.550 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_8 /tmp/tmp.TfDn6BcQ3N/cluster1-mirror-image_8.status 2023-08-26T13:52:48.550 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster1 2023-08-26T13:52:48.550 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T13:52:48.551 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=image_8 2023-08-26T13:52:48.551 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local log=/tmp/tmp.TfDn6BcQ3N/cluster1-mirror-image_8.status 2023-08-26T13:52:48.551 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_8 --format xml 2023-08-26T13:52:48.551 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-08-26T13:52:48.598 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ true 2023-08-26T13:52:48.599 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-08-26T13:52:48.600 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:-:1.1: Document is empty 2023-08-26T13:52:48.600 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr: 2023-08-26T13:52:48.600 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:^ 2023-08-26T13:52:48.601 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local primary_snapshot_id= 2023-08-26T13:52:48.601 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ test 133 = '' 2023-08-26T13:52:48.601 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-08-26T13:52:48.601 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ sleep 0.4 2023-08-26T13:52:49.001 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_8 /tmp/tmp.TfDn6BcQ3N/cluster1-mirror-image_8.status 2023-08-26T13:52:49.002 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster1 2023-08-26T13:52:49.002 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T13:52:49.002 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=image_8 2023-08-26T13:52:49.002 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local log=/tmp/tmp.TfDn6BcQ3N/cluster1-mirror-image_8.status 2023-08-26T13:52:49.003 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_8 --format xml 2023-08-26T13:52:49.003 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-08-26T13:52:49.050 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ true 2023-08-26T13:52:49.050 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-08-26T13:52:49.051 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:-:1.1: Document is empty 2023-08-26T13:52:49.051 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr: 2023-08-26T13:52:49.052 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:^ 2023-08-26T13:52:49.052 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local primary_snapshot_id= 2023-08-26T13:52:49.053 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ test 133 = '' 2023-08-26T13:52:49.053 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-08-26T13:52:49.053 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ sleep 0.8 2023-08-26T13:52:49.853 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_8 /tmp/tmp.TfDn6BcQ3N/cluster1-mirror-image_8.status 2023-08-26T13:52:49.853 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster1 2023-08-26T13:52:49.854 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T13:52:49.854 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=image_8 2023-08-26T13:52:49.854 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local log=/tmp/tmp.TfDn6BcQ3N/cluster1-mirror-image_8.status 2023-08-26T13:52:49.854 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_8 --format xml 2023-08-26T13:52:49.854 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-08-26T13:52:49.897 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-08-26T13:52:49.899 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local primary_snapshot_id=133 2023-08-26T13:52:49.899 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ test 133 = 133 2023-08-26T13:52:49.900 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ return 0 2023-08-26T13:52:49.900 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ wait_for_snap_present cluster1 mirror image_8 snap 2023-08-26T13:52:49.900 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster1 2023-08-26T13:52:49.900 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T13:52:49.900 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=image_8 2023-08-26T13:52:49.901 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local snap_name=snap 2023-08-26T13:52:49.901 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local s 2023-08-26T13:52:49.901 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2023-08-26T13:52:49.901 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ sleep 1 2023-08-26T13:52:50.900 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rbd --cluster cluster1 info mirror/image_8@snap 2023-08-26T13:52:50.945 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:rbd image 'image_8': 2023-08-26T13:52:50.945 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: size 128 MiB in 32 objects 2023-08-26T13:52:50.945 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: order 22 (4 MiB objects) 2023-08-26T13:52:50.945 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: snapshot_count: 2 2023-08-26T13:52:50.946 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: id: 12b6474329c 2023-08-26T13:52:50.946 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: block_name_prefix: rbd_data.12b6474329c 2023-08-26T13:52:50.946 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: format: 2 2023-08-26T13:52:50.946 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: features: layering, exclusive-lock, non-primary 2023-08-26T13:52:50.946 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: op_features: 2023-08-26T13:52:50.947 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: flags: 2023-08-26T13:52:50.947 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: create_timestamp: Sat Aug 26 13:43:57 2023 2023-08-26T13:52:50.947 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: access_timestamp: Sat Aug 26 13:43:57 2023 2023-08-26T13:52:50.947 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: modify_timestamp: Sat Aug 26 13:43:57 2023 2023-08-26T13:52:50.947 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: protected: False 2023-08-26T13:52:50.948 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: mirroring state: enabled 2023-08-26T13:52:50.948 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: mirroring mode: snapshot 2023-08-26T13:52:50.948 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: mirroring global id: 0e8ffb4e-bbd1-41aa-b41a-6b2aebda2762 2023-08-26T13:52:50.948 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: mirroring primary: false 2023-08-26T13:52:50.949 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ return 0 2023-08-26T13:52:50.949 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ compare_image_snaps mirror image_8 snap 2023-08-26T13:52:50.950 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T13:52:50.950 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=image_8 2023-08-26T13:52:50.950 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local snap_name=snap 2023-08-26T13:52:50.950 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local ret=0 2023-08-26T13:52:50.950 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local rmt_export=/tmp/tmp.TfDn6BcQ3N/cluster2-mirror-image_8.export 2023-08-26T13:52:50.951 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local loc_export=/tmp/tmp.TfDn6BcQ3N/cluster1-mirror-image_8.export 2023-08-26T13:52:50.951 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rm -f /tmp/tmp.TfDn6BcQ3N/cluster2-mirror-image_8.export /tmp/tmp.TfDn6BcQ3N/cluster1-mirror-image_8.export 2023-08-26T13:52:50.951 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rbd --cluster cluster2 -p mirror export image_8@snap /tmp/tmp.TfDn6BcQ3N/cluster2-mirror-image_8.export 2023-08-26T13:52:51.116 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-08-26T13:52:51.121 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rbd --cluster cluster1 -p mirror export image_8@snap /tmp/tmp.TfDn6BcQ3N/cluster1-mirror-image_8.export 2023-08-26T13:52:51.357 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-08-26T13:52:51.363 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ cmp /tmp/tmp.TfDn6BcQ3N/cluster2-mirror-image_8.export /tmp/tmp.TfDn6BcQ3N/cluster1-mirror-image_8.export 2023-08-26T13:52:51.419 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rm -f /tmp/tmp.TfDn6BcQ3N/cluster2-mirror-image_8.export /tmp/tmp.TfDn6BcQ3N/cluster1-mirror-image_8.export 2023-08-26T13:52:51.456 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ return 0 2023-08-26T13:52:51.456 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-08-26T13:52:51.456 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ image=image_9 2023-08-26T13:52:51.456 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ create_snap cluster2 mirror image_9 snap 2023-08-26T13:52:51.456 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster2 2023-08-26T13:52:51.457 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T13:52:51.457 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=image_9 2023-08-26T13:52:51.457 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local snap_name=snap 2023-08-26T13:52:51.457 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rbd --cluster cluster2 -p mirror snap create image_9@snap --debug-rbd=20 --debug-journaler=20 2023-08-26T13:52:53.305 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ wait_for_image_replay_started cluster1 mirror image_9 2023-08-26T13:52:53.305 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster1 2023-08-26T13:52:53.305 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T13:52:53.305 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=image_9 2023-08-26T13:52:53.306 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ wait_for_image_replay_state cluster1 mirror image_9 started 2023-08-26T13:52:53.306 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster1 2023-08-26T13:52:53.306 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T13:52:53.306 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=image_9 2023-08-26T13:52:53.306 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local state=started 2023-08-26T13:52:53.306 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local s 2023-08-26T13:52:53.307 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-08-26T13:52:53.307 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ sleep 1 2023-08-26T13:52:54.305 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ test_image_replay_state cluster1 mirror image_9 started 2023-08-26T13:52:54.306 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster1 2023-08-26T13:52:54.306 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T13:52:54.306 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=image_9 2023-08-26T13:52:54.306 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local test_state=started 2023-08-26T13:52:54.307 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local status_result 2023-08-26T13:52:54.307 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local current_state=stopped 2023-08-26T13:52:54.310 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_9 2023-08-26T13:52:54.311 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local cluster_instance=cluster1 2023-08-26T13:52:54.311 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ shift 2023-08-26T13:52:54.311 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ grep -i state 2023-08-26T13:52:54.311 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local cluster=cluster1 2023-08-26T13:52:54.311 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local instance=cluster1 2023-08-26T13:52:54.312 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local loop_instance 2023-08-26T13:52:54.312 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-08-26T13:52:54.312 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ sleep 0 2023-08-26T13:52:54.312 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ '[' cluster1 '!=' cluster1 ']' 2023-08-26T13:52:54.312 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ seq 0 3 2023-08-26T13:52:54.313 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-08-26T13:52:54.313 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_9 2023-08-26T13:52:54.313 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local cluster=cluster1:0 2023-08-26T13:52:54.313 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ shift 2023-08-26T13:52:54.313 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local instance 2023-08-26T13:52:54.313 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ set_cluster_instance cluster1:0 cluster instance 2023-08-26T13:52:54.314 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local val=cluster1:0 2023-08-26T13:52:54.314 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local cluster_var_name=cluster 2023-08-26T13:52:54.314 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local instance_var_name=instance 2023-08-26T13:52:54.314 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ cluster=cluster1 2023-08-26T13:52:54.314 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ instance=0 2023-08-26T13:52:54.314 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ '[' 0 = cluster1:0 ']' 2023-08-26T13:52:54.315 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ eval cluster=cluster1 2023-08-26T13:52:54.315 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ cluster=cluster1 2023-08-26T13:52:54.315 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ eval instance=0 2023-08-26T13:52:54.315 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ instance=0 2023-08-26T13:52:54.315 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ daemon_asok_file cluster1:0 cluster1 2023-08-26T13:52:54.316 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ local local_cluster=cluster1:0 2023-08-26T13:52:54.316 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ local cluster=cluster1 2023-08-26T13:52:54.316 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ local instance 2023-08-26T13:52:54.316 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2023-08-26T13:52:54.316 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ local val=cluster1:0 2023-08-26T13:52:54.317 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ local cluster_var_name=local_cluster 2023-08-26T13:52:54.317 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ local instance_var_name=instance 2023-08-26T13:52:54.317 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ cluster=cluster1 2023-08-26T13:52:54.317 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ instance=0 2023-08-26T13:52:54.317 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ '[' 0 = cluster1:0 ']' 2023-08-26T13:52:54.317 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ eval local_cluster=cluster1 2023-08-26T13:52:54.318 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++++ local_cluster=cluster1 2023-08-26T13:52:54.318 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ eval instance=0 2023-08-26T13:52:54.318 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++++ instance=0 2023-08-26T13:52:54.318 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2023-08-26T13:52:54.331 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-08-26T13:52:54.331 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-08-26T13:52:54.332 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-08-26T13:52:54.332 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_9 2023-08-26T13:52:54.440 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:Invalid command: unused arguments: ['mirror/image_9'] 2023-08-26T13:52:54.441 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:rbd mirror status : get status for rbd mirror 2023-08-26T13:52:54.441 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:admin_socket: invalid command 2023-08-26T13:52:54.443 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-08-26T13:52:54.444 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_9 2023-08-26T13:52:54.444 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local cluster=cluster1:1 2023-08-26T13:52:54.444 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ shift 2023-08-26T13:52:54.444 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local instance 2023-08-26T13:52:54.444 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ set_cluster_instance cluster1:1 cluster instance 2023-08-26T13:52:54.445 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local val=cluster1:1 2023-08-26T13:52:54.445 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local cluster_var_name=cluster 2023-08-26T13:52:54.445 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local instance_var_name=instance 2023-08-26T13:52:54.445 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ cluster=cluster1 2023-08-26T13:52:54.445 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ instance=1 2023-08-26T13:52:54.446 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ '[' 1 = cluster1:1 ']' 2023-08-26T13:52:54.446 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ eval cluster=cluster1 2023-08-26T13:52:54.446 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ cluster=cluster1 2023-08-26T13:52:54.446 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ eval instance=1 2023-08-26T13:52:54.446 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ instance=1 2023-08-26T13:52:54.447 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ daemon_asok_file cluster1:1 cluster1 2023-08-26T13:52:54.447 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ local local_cluster=cluster1:1 2023-08-26T13:52:54.447 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ local cluster=cluster1 2023-08-26T13:52:54.447 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ local instance 2023-08-26T13:52:54.447 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2023-08-26T13:52:54.447 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ local val=cluster1:1 2023-08-26T13:52:54.448 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ local cluster_var_name=local_cluster 2023-08-26T13:52:54.448 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ local instance_var_name=instance 2023-08-26T13:52:54.448 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ cluster=cluster1 2023-08-26T13:52:54.448 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ instance=1 2023-08-26T13:52:54.448 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ '[' 1 = cluster1:1 ']' 2023-08-26T13:52:54.449 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ eval local_cluster=cluster1 2023-08-26T13:52:54.449 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++++ local_cluster=cluster1 2023-08-26T13:52:54.449 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ eval instance=1 2023-08-26T13:52:54.449 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++++ instance=1 2023-08-26T13:52:54.449 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2023-08-26T13:52:54.463 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-08-26T13:52:54.463 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-08-26T13:52:54.464 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-08-26T13:52:54.464 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_9 2023-08-26T13:52:54.570 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:Invalid command: unused arguments: ['mirror/image_9'] 2023-08-26T13:52:54.570 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:rbd mirror status : get status for rbd mirror 2023-08-26T13:52:54.570 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:admin_socket: invalid command 2023-08-26T13:52:54.573 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-08-26T13:52:54.573 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/image_9 2023-08-26T13:52:54.574 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local cluster=cluster1:2 2023-08-26T13:52:54.574 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ shift 2023-08-26T13:52:54.574 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local instance 2023-08-26T13:52:54.574 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ set_cluster_instance cluster1:2 cluster instance 2023-08-26T13:52:54.574 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local val=cluster1:2 2023-08-26T13:52:54.574 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local cluster_var_name=cluster 2023-08-26T13:52:54.575 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local instance_var_name=instance 2023-08-26T13:52:54.575 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ cluster=cluster1 2023-08-26T13:52:54.575 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ instance=2 2023-08-26T13:52:54.575 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ '[' 2 = cluster1:2 ']' 2023-08-26T13:52:54.575 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ eval cluster=cluster1 2023-08-26T13:52:54.576 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ cluster=cluster1 2023-08-26T13:52:54.576 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ eval instance=2 2023-08-26T13:52:54.576 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ instance=2 2023-08-26T13:52:54.576 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ daemon_asok_file cluster1:2 cluster1 2023-08-26T13:52:54.576 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ local local_cluster=cluster1:2 2023-08-26T13:52:54.577 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ local cluster=cluster1 2023-08-26T13:52:54.577 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ local instance 2023-08-26T13:52:54.577 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2023-08-26T13:52:54.577 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ local val=cluster1:2 2023-08-26T13:52:54.577 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ local cluster_var_name=local_cluster 2023-08-26T13:52:54.577 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ local instance_var_name=instance 2023-08-26T13:52:54.578 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ cluster=cluster1 2023-08-26T13:52:54.578 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ instance=2 2023-08-26T13:52:54.578 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ '[' 2 = cluster1:2 ']' 2023-08-26T13:52:54.578 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ eval local_cluster=cluster1 2023-08-26T13:52:54.578 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++++ local_cluster=cluster1 2023-08-26T13:52:54.579 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ eval instance=2 2023-08-26T13:52:54.579 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++++ instance=2 2023-08-26T13:52:54.579 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2023-08-26T13:52:54.591 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-08-26T13:52:54.592 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-08-26T13:52:54.592 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-08-26T13:52:54.592 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/image_9 2023-08-26T13:52:54.706 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ return 0 2023-08-26T13:52:54.707 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ status_result=' "state": "Replaying"' 2023-08-26T13:52:54.707 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ echo ' "state": "Replaying"' 2023-08-26T13:52:54.707 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ grep -i Replaying 2023-08-26T13:52:54.708 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: "state": "Replaying" 2023-08-26T13:52:54.710 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ current_state=started 2023-08-26T13:52:54.710 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ test started = started 2023-08-26T13:52:54.710 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ return 0 2023-08-26T13:52:54.711 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_9 2023-08-26T13:52:54.711 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local local_cluster=cluster1 2023-08-26T13:52:54.711 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster2 2023-08-26T13:52:54.711 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T13:52:54.711 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=image_9 2023-08-26T13:52:54.712 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ '[' snapshot = journal ']' 2023-08-26T13:52:54.712 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ '[' snapshot = snapshot ']' 2023-08-26T13:52:54.712 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_9 2023-08-26T13:52:54.712 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local local_cluster=cluster1 2023-08-26T13:52:54.712 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster2 2023-08-26T13:52:54.713 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T13:52:54.713 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=image_9 2023-08-26T13:52:54.713 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ mkfname cluster2-mirror-image_9.status 2023-08-26T13:52:54.713 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ echo cluster2-mirror-image_9.status 2023-08-26T13:52:54.713 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ sed -e 's|[/ ]|_|g' 2023-08-26T13:52:54.713 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local status_log=/tmp/tmp.TfDn6BcQ3N/cluster2-mirror-image_9.status 2023-08-26T13:52:54.714 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ mkfname cluster1-mirror-image_9.status 2023-08-26T13:52:54.714 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ echo cluster1-mirror-image_9.status 2023-08-26T13:52:54.714 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ sed -e 's|[/ ]|_|g' 2023-08-26T13:52:54.715 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local local_status_log=/tmp/tmp.TfDn6BcQ3N/cluster1-mirror-image_9.status 2023-08-26T13:52:54.715 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ mirror_image_snapshot cluster2 mirror image_9 2023-08-26T13:52:54.715 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster2 2023-08-26T13:52:54.715 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T13:52:54.716 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=image_9 2023-08-26T13:52:54.716 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_9 2023-08-26T13:52:55.307 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:Snapshot ID: 135 2023-08-26T13:52:55.315 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_9 /tmp/tmp.TfDn6BcQ3N/cluster2-mirror-image_9.status 2023-08-26T13:52:55.315 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster2 2023-08-26T13:52:55.315 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T13:52:55.316 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=image_9 2023-08-26T13:52:55.316 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local log=/tmp/tmp.TfDn6BcQ3N/cluster2-mirror-image_9.status 2023-08-26T13:52:55.316 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_9 --format xml 2023-08-26T13:52:55.316 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-08-26T13:52:55.357 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ xmlstarlet sel -t -v //snapshot/id 2023-08-26T13:52:55.359 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local snapshot_id=135 2023-08-26T13:52:55.359 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ true 2023-08-26T13:52:55.359 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-08-26T13:52:55.360 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ sleep 0.2 2023-08-26T13:52:55.560 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_9 /tmp/tmp.TfDn6BcQ3N/cluster1-mirror-image_9.status 2023-08-26T13:52:55.560 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster1 2023-08-26T13:52:55.561 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T13:52:55.561 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=image_9 2023-08-26T13:52:55.561 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local log=/tmp/tmp.TfDn6BcQ3N/cluster1-mirror-image_9.status 2023-08-26T13:52:55.561 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_9 --format xml 2023-08-26T13:52:55.561 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-08-26T13:52:55.607 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-08-26T13:52:55.608 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local primary_snapshot_id=76 2023-08-26T13:52:55.609 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ test 135 = 76 2023-08-26T13:52:55.609 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-08-26T13:52:55.609 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ sleep 0.4 2023-08-26T13:52:56.010 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_9 /tmp/tmp.TfDn6BcQ3N/cluster1-mirror-image_9.status 2023-08-26T13:52:56.010 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster1 2023-08-26T13:52:56.010 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T13:52:56.010 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=image_9 2023-08-26T13:52:56.011 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local log=/tmp/tmp.TfDn6BcQ3N/cluster1-mirror-image_9.status 2023-08-26T13:52:56.011 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_9 --format xml 2023-08-26T13:52:56.011 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-08-26T13:52:56.056 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-08-26T13:52:56.058 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local primary_snapshot_id=76 2023-08-26T13:52:56.058 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ test 135 = 76 2023-08-26T13:52:56.059 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-08-26T13:52:56.059 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ sleep 0.8 2023-08-26T13:52:56.860 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_9 /tmp/tmp.TfDn6BcQ3N/cluster1-mirror-image_9.status 2023-08-26T13:52:56.860 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster1 2023-08-26T13:52:56.860 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T13:52:56.860 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=image_9 2023-08-26T13:52:56.861 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local log=/tmp/tmp.TfDn6BcQ3N/cluster1-mirror-image_9.status 2023-08-26T13:52:56.861 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_9 --format xml 2023-08-26T13:52:56.861 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-08-26T13:52:56.908 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ true 2023-08-26T13:52:56.908 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-08-26T13:52:56.909 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:-:1.1: Document is empty 2023-08-26T13:52:56.909 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr: 2023-08-26T13:52:56.909 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:^ 2023-08-26T13:52:56.910 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local primary_snapshot_id= 2023-08-26T13:52:56.910 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ test 135 = '' 2023-08-26T13:52:56.911 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-08-26T13:52:56.911 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ sleep 1.6 2023-08-26T13:52:58.511 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_9 /tmp/tmp.TfDn6BcQ3N/cluster1-mirror-image_9.status 2023-08-26T13:52:58.511 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster1 2023-08-26T13:52:58.512 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T13:52:58.512 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=image_9 2023-08-26T13:52:58.512 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local log=/tmp/tmp.TfDn6BcQ3N/cluster1-mirror-image_9.status 2023-08-26T13:52:58.512 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_9 --format xml 2023-08-26T13:52:58.512 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-08-26T13:52:58.519 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.0 with skip (0.8107682778928238) > weight (0.25) 2023-08-26T13:52:58.519 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.1 with skip (0.4728425557550848) > weight (0.25) 2023-08-26T13:52:58.519 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.2 with skip (0.3044048303631828) > weight (0.25) 2023-08-26T13:52:58.519 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.3 with skip (0.4354660166899018) > weight (0.25) 2023-08-26T13:52:58.520 INFO:tasks.rbd_mirror_thrash:waiting for 81 secs before thrashing 2023-08-26T13:52:58.562 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-08-26T13:52:58.564 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local primary_snapshot_id=135 2023-08-26T13:52:58.564 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ test 135 = 135 2023-08-26T13:52:58.565 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ return 0 2023-08-26T13:52:58.565 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ wait_for_snap_present cluster1 mirror image_9 snap 2023-08-26T13:52:58.565 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster1 2023-08-26T13:52:58.565 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T13:52:58.565 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=image_9 2023-08-26T13:52:58.566 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local snap_name=snap 2023-08-26T13:52:58.566 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local s 2023-08-26T13:52:58.566 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2023-08-26T13:52:58.566 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ sleep 1 2023-08-26T13:52:59.565 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rbd --cluster cluster1 info mirror/image_9@snap 2023-08-26T13:52:59.611 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:rbd image 'image_9': 2023-08-26T13:52:59.611 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: size 128 MiB in 32 objects 2023-08-26T13:52:59.611 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: order 22 (4 MiB objects) 2023-08-26T13:52:59.612 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: snapshot_count: 2 2023-08-26T13:52:59.612 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: id: 122b82bc64c5 2023-08-26T13:52:59.612 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: block_name_prefix: rbd_data.122b82bc64c5 2023-08-26T13:52:59.612 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: format: 2 2023-08-26T13:52:59.612 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: features: layering, exclusive-lock, non-primary 2023-08-26T13:52:59.613 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: op_features: 2023-08-26T13:52:59.613 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: flags: 2023-08-26T13:52:59.613 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: create_timestamp: Sat Aug 26 13:44:25 2023 2023-08-26T13:52:59.613 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: access_timestamp: Sat Aug 26 13:44:25 2023 2023-08-26T13:52:59.613 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: modify_timestamp: Sat Aug 26 13:44:25 2023 2023-08-26T13:52:59.614 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: protected: False 2023-08-26T13:52:59.614 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: mirroring state: enabled 2023-08-26T13:52:59.614 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: mirroring mode: snapshot 2023-08-26T13:52:59.614 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: mirroring global id: 680b775a-fecc-478c-ab5f-9dc6f2f67724 2023-08-26T13:52:59.614 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: mirroring primary: false 2023-08-26T13:52:59.615 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ return 0 2023-08-26T13:52:59.616 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ compare_image_snaps mirror image_9 snap 2023-08-26T13:52:59.616 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T13:52:59.616 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=image_9 2023-08-26T13:52:59.616 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local snap_name=snap 2023-08-26T13:52:59.617 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local ret=0 2023-08-26T13:52:59.617 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local rmt_export=/tmp/tmp.TfDn6BcQ3N/cluster2-mirror-image_9.export 2023-08-26T13:52:59.617 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local loc_export=/tmp/tmp.TfDn6BcQ3N/cluster1-mirror-image_9.export 2023-08-26T13:52:59.617 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rm -f /tmp/tmp.TfDn6BcQ3N/cluster2-mirror-image_9.export /tmp/tmp.TfDn6BcQ3N/cluster1-mirror-image_9.export 2023-08-26T13:52:59.617 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rbd --cluster cluster2 -p mirror export image_9@snap /tmp/tmp.TfDn6BcQ3N/cluster2-mirror-image_9.export 2023-08-26T13:52:59.803 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-08-26T13:52:59.810 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rbd --cluster cluster1 -p mirror export image_9@snap /tmp/tmp.TfDn6BcQ3N/cluster1-mirror-image_9.export 2023-08-26T13:53:00.028 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-08-26T13:53:00.033 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ cmp /tmp/tmp.TfDn6BcQ3N/cluster2-mirror-image_9.export /tmp/tmp.TfDn6BcQ3N/cluster1-mirror-image_9.export 2023-08-26T13:53:00.090 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rm -f /tmp/tmp.TfDn6BcQ3N/cluster2-mirror-image_9.export /tmp/tmp.TfDn6BcQ3N/cluster1-mirror-image_9.export 2023-08-26T13:53:00.127 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ return 0 2023-08-26T13:53:00.128 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-08-26T13:53:00.128 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ image=image_10 2023-08-26T13:53:00.128 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ create_snap cluster2 mirror image_10 snap 2023-08-26T13:53:00.128 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster2 2023-08-26T13:53:00.128 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T13:53:00.129 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=image_10 2023-08-26T13:53:00.129 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local snap_name=snap 2023-08-26T13:53:00.129 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rbd --cluster cluster2 -p mirror snap create image_10@snap --debug-rbd=20 --debug-journaler=20 2023-08-26T13:53:01.321 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ wait_for_image_replay_started cluster1 mirror image_10 2023-08-26T13:53:01.322 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster1 2023-08-26T13:53:01.322 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T13:53:01.322 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=image_10 2023-08-26T13:53:01.322 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ wait_for_image_replay_state cluster1 mirror image_10 started 2023-08-26T13:53:01.323 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster1 2023-08-26T13:53:01.323 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T13:53:01.323 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=image_10 2023-08-26T13:53:01.323 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local state=started 2023-08-26T13:53:01.323 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local s 2023-08-26T13:53:01.324 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-08-26T13:53:01.324 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ sleep 1 2023-08-26T13:53:02.322 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ test_image_replay_state cluster1 mirror image_10 started 2023-08-26T13:53:02.323 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster1 2023-08-26T13:53:02.323 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T13:53:02.323 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=image_10 2023-08-26T13:53:02.323 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local test_state=started 2023-08-26T13:53:02.324 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local status_result 2023-08-26T13:53:02.324 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local current_state=stopped 2023-08-26T13:53:02.326 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_10 2023-08-26T13:53:02.327 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local cluster_instance=cluster1 2023-08-26T13:53:02.327 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ grep -i state 2023-08-26T13:53:02.327 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ shift 2023-08-26T13:53:02.327 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local cluster=cluster1 2023-08-26T13:53:02.327 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local instance=cluster1 2023-08-26T13:53:02.328 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local loop_instance 2023-08-26T13:53:02.328 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-08-26T13:53:02.328 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ sleep 0 2023-08-26T13:53:02.328 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ '[' cluster1 '!=' cluster1 ']' 2023-08-26T13:53:02.328 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ seq 0 3 2023-08-26T13:53:02.329 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-08-26T13:53:02.329 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_10 2023-08-26T13:53:02.329 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local cluster=cluster1:0 2023-08-26T13:53:02.329 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ shift 2023-08-26T13:53:02.329 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local instance 2023-08-26T13:53:02.330 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ set_cluster_instance cluster1:0 cluster instance 2023-08-26T13:53:02.330 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local val=cluster1:0 2023-08-26T13:53:02.330 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local cluster_var_name=cluster 2023-08-26T13:53:02.330 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local instance_var_name=instance 2023-08-26T13:53:02.330 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ cluster=cluster1 2023-08-26T13:53:02.331 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ instance=0 2023-08-26T13:53:02.331 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ '[' 0 = cluster1:0 ']' 2023-08-26T13:53:02.331 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ eval cluster=cluster1 2023-08-26T13:53:02.331 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ cluster=cluster1 2023-08-26T13:53:02.331 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ eval instance=0 2023-08-26T13:53:02.331 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ instance=0 2023-08-26T13:53:02.332 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ daemon_asok_file cluster1:0 cluster1 2023-08-26T13:53:02.333 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ local local_cluster=cluster1:0 2023-08-26T13:53:02.333 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ local cluster=cluster1 2023-08-26T13:53:02.333 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ local instance 2023-08-26T13:53:02.334 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2023-08-26T13:53:02.334 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ local val=cluster1:0 2023-08-26T13:53:02.334 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ local cluster_var_name=local_cluster 2023-08-26T13:53:02.334 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ local instance_var_name=instance 2023-08-26T13:53:02.334 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ cluster=cluster1 2023-08-26T13:53:02.335 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ instance=0 2023-08-26T13:53:02.335 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ '[' 0 = cluster1:0 ']' 2023-08-26T13:53:02.335 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ eval local_cluster=cluster1 2023-08-26T13:53:02.335 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++++ local_cluster=cluster1 2023-08-26T13:53:02.335 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ eval instance=0 2023-08-26T13:53:02.335 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++++ instance=0 2023-08-26T13:53:02.336 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2023-08-26T13:53:02.349 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-08-26T13:53:02.350 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-08-26T13:53:02.350 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-08-26T13:53:02.350 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_10 2023-08-26T13:53:02.471 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ return 0 2023-08-26T13:53:02.472 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ status_result=' "state": "Replaying"' 2023-08-26T13:53:02.472 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ echo ' "state": "Replaying"' 2023-08-26T13:53:02.473 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ grep -i Replaying 2023-08-26T13:53:02.473 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: "state": "Replaying" 2023-08-26T13:53:02.475 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ current_state=started 2023-08-26T13:53:02.476 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ test started = started 2023-08-26T13:53:02.476 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ return 0 2023-08-26T13:53:02.476 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_10 2023-08-26T13:53:02.476 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local local_cluster=cluster1 2023-08-26T13:53:02.477 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster2 2023-08-26T13:53:02.477 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T13:53:02.477 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=image_10 2023-08-26T13:53:02.477 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ '[' snapshot = journal ']' 2023-08-26T13:53:02.477 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ '[' snapshot = snapshot ']' 2023-08-26T13:53:02.478 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_10 2023-08-26T13:53:02.478 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local local_cluster=cluster1 2023-08-26T13:53:02.478 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster2 2023-08-26T13:53:02.478 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T13:53:02.478 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=image_10 2023-08-26T13:53:02.478 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ mkfname cluster2-mirror-image_10.status 2023-08-26T13:53:02.479 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ echo cluster2-mirror-image_10.status 2023-08-26T13:53:02.479 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ sed -e 's|[/ ]|_|g' 2023-08-26T13:53:02.479 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local status_log=/tmp/tmp.TfDn6BcQ3N/cluster2-mirror-image_10.status 2023-08-26T13:53:02.479 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ mkfname cluster1-mirror-image_10.status 2023-08-26T13:53:02.479 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ echo cluster1-mirror-image_10.status 2023-08-26T13:53:02.480 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ sed -e 's|[/ ]|_|g' 2023-08-26T13:53:02.480 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local local_status_log=/tmp/tmp.TfDn6BcQ3N/cluster1-mirror-image_10.status 2023-08-26T13:53:02.481 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ mirror_image_snapshot cluster2 mirror image_10 2023-08-26T13:53:02.481 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster2 2023-08-26T13:53:02.481 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T13:53:02.481 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=image_10 2023-08-26T13:53:02.481 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_10 2023-08-26T13:53:03.331 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:Snapshot ID: 137 2023-08-26T13:53:03.341 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_10 /tmp/tmp.TfDn6BcQ3N/cluster2-mirror-image_10.status 2023-08-26T13:53:03.342 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster2 2023-08-26T13:53:03.342 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T13:53:03.342 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=image_10 2023-08-26T13:53:03.342 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local log=/tmp/tmp.TfDn6BcQ3N/cluster2-mirror-image_10.status 2023-08-26T13:53:03.342 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_10 --format xml 2023-08-26T13:53:03.343 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-08-26T13:53:03.387 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ xmlstarlet sel -t -v //snapshot/id 2023-08-26T13:53:03.389 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local snapshot_id=137 2023-08-26T13:53:03.389 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ true 2023-08-26T13:53:03.389 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-08-26T13:53:03.390 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ sleep 0.2 2023-08-26T13:53:03.590 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_10 /tmp/tmp.TfDn6BcQ3N/cluster1-mirror-image_10.status 2023-08-26T13:53:03.591 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster1 2023-08-26T13:53:03.591 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T13:53:03.591 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=image_10 2023-08-26T13:53:03.591 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local log=/tmp/tmp.TfDn6BcQ3N/cluster1-mirror-image_10.status 2023-08-26T13:53:03.592 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_10 --format xml 2023-08-26T13:53:03.592 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-08-26T13:53:03.642 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ true 2023-08-26T13:53:03.643 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-08-26T13:53:03.644 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:-:1.1: Document is empty 2023-08-26T13:53:03.644 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr: 2023-08-26T13:53:03.645 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:^ 2023-08-26T13:53:03.645 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local primary_snapshot_id= 2023-08-26T13:53:03.646 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ test 137 = '' 2023-08-26T13:53:03.646 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-08-26T13:53:03.646 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ sleep 0.4 2023-08-26T13:53:04.046 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_10 /tmp/tmp.TfDn6BcQ3N/cluster1-mirror-image_10.status 2023-08-26T13:53:04.047 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster1 2023-08-26T13:53:04.047 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T13:53:04.047 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=image_10 2023-08-26T13:53:04.047 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local log=/tmp/tmp.TfDn6BcQ3N/cluster1-mirror-image_10.status 2023-08-26T13:53:04.048 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_10 --format xml 2023-08-26T13:53:04.048 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-08-26T13:53:04.096 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ true 2023-08-26T13:53:04.096 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-08-26T13:53:04.097 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:-:1.1: Document is empty 2023-08-26T13:53:04.097 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr: 2023-08-26T13:53:04.097 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:^ 2023-08-26T13:53:04.098 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local primary_snapshot_id= 2023-08-26T13:53:04.098 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ test 137 = '' 2023-08-26T13:53:04.098 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-08-26T13:53:04.099 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ sleep 0.8 2023-08-26T13:53:04.899 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_10 /tmp/tmp.TfDn6BcQ3N/cluster1-mirror-image_10.status 2023-08-26T13:53:04.900 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster1 2023-08-26T13:53:04.900 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T13:53:04.900 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=image_10 2023-08-26T13:53:04.900 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local log=/tmp/tmp.TfDn6BcQ3N/cluster1-mirror-image_10.status 2023-08-26T13:53:04.900 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_10 --format xml 2023-08-26T13:53:04.901 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-08-26T13:53:04.945 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-08-26T13:53:04.947 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local primary_snapshot_id=137 2023-08-26T13:53:04.947 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ test 137 = 137 2023-08-26T13:53:04.947 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ return 0 2023-08-26T13:53:04.948 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ wait_for_snap_present cluster1 mirror image_10 snap 2023-08-26T13:53:04.948 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster1 2023-08-26T13:53:04.948 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T13:53:04.948 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=image_10 2023-08-26T13:53:04.948 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local snap_name=snap 2023-08-26T13:53:04.949 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local s 2023-08-26T13:53:04.949 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2023-08-26T13:53:04.949 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ sleep 1 2023-08-26T13:53:05.948 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rbd --cluster cluster1 info mirror/image_10@snap 2023-08-26T13:53:05.993 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:rbd image 'image_10': 2023-08-26T13:53:05.993 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: size 128 MiB in 32 objects 2023-08-26T13:53:05.993 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: order 22 (4 MiB objects) 2023-08-26T13:53:05.993 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: snapshot_count: 2 2023-08-26T13:53:05.994 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: id: 12b6474d056f 2023-08-26T13:53:05.994 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: block_name_prefix: rbd_data.12b6474d056f 2023-08-26T13:53:05.994 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: format: 2 2023-08-26T13:53:05.994 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: features: layering, exclusive-lock, non-primary 2023-08-26T13:53:05.994 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: op_features: 2023-08-26T13:53:05.995 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: flags: 2023-08-26T13:53:05.995 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: create_timestamp: Sat Aug 26 13:44:16 2023 2023-08-26T13:53:05.995 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: access_timestamp: Sat Aug 26 13:44:16 2023 2023-08-26T13:53:05.995 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: modify_timestamp: Sat Aug 26 13:44:16 2023 2023-08-26T13:53:05.995 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: protected: False 2023-08-26T13:53:05.996 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: mirroring state: enabled 2023-08-26T13:53:05.996 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: mirroring mode: snapshot 2023-08-26T13:53:05.996 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: mirroring global id: 83466149-7311-4c4c-898e-a67f6a0dbe7c 2023-08-26T13:53:05.996 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: mirroring primary: false 2023-08-26T13:53:05.997 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ return 0 2023-08-26T13:53:05.998 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ compare_image_snaps mirror image_10 snap 2023-08-26T13:53:05.998 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T13:53:05.998 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=image_10 2023-08-26T13:53:05.998 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local snap_name=snap 2023-08-26T13:53:05.998 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local ret=0 2023-08-26T13:53:05.999 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local rmt_export=/tmp/tmp.TfDn6BcQ3N/cluster2-mirror-image_10.export 2023-08-26T13:53:05.999 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local loc_export=/tmp/tmp.TfDn6BcQ3N/cluster1-mirror-image_10.export 2023-08-26T13:53:05.999 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rm -f /tmp/tmp.TfDn6BcQ3N/cluster2-mirror-image_10.export /tmp/tmp.TfDn6BcQ3N/cluster1-mirror-image_10.export 2023-08-26T13:53:05.999 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rbd --cluster cluster2 -p mirror export image_10@snap /tmp/tmp.TfDn6BcQ3N/cluster2-mirror-image_10.export 2023-08-26T13:53:06.164 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-08-26T13:53:06.169 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rbd --cluster cluster1 -p mirror export image_10@snap /tmp/tmp.TfDn6BcQ3N/cluster1-mirror-image_10.export 2023-08-26T13:53:06.419 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-08-26T13:53:06.425 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ cmp /tmp/tmp.TfDn6BcQ3N/cluster2-mirror-image_10.export /tmp/tmp.TfDn6BcQ3N/cluster1-mirror-image_10.export 2023-08-26T13:53:06.480 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rm -f /tmp/tmp.TfDn6BcQ3N/cluster2-mirror-image_10.export /tmp/tmp.TfDn6BcQ3N/cluster1-mirror-image_10.export 2023-08-26T13:53:06.517 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ return 0 2023-08-26T13:53:06.518 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-08-26T13:53:06.518 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ image=image_11 2023-08-26T13:53:06.518 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ create_snap cluster2 mirror image_11 snap 2023-08-26T13:53:06.518 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster2 2023-08-26T13:53:06.518 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T13:53:06.519 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=image_11 2023-08-26T13:53:06.519 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local snap_name=snap 2023-08-26T13:53:06.519 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rbd --cluster cluster2 -p mirror snap create image_11@snap --debug-rbd=20 --debug-journaler=20 2023-08-26T13:53:08.342 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ wait_for_image_replay_started cluster1 mirror image_11 2023-08-26T13:53:08.342 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster1 2023-08-26T13:53:08.342 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T13:53:08.342 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=image_11 2023-08-26T13:53:08.342 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ wait_for_image_replay_state cluster1 mirror image_11 started 2023-08-26T13:53:08.343 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster1 2023-08-26T13:53:08.343 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T13:53:08.343 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=image_11 2023-08-26T13:53:08.343 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local state=started 2023-08-26T13:53:08.343 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local s 2023-08-26T13:53:08.344 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-08-26T13:53:08.344 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ sleep 1 2023-08-26T13:53:09.344 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ test_image_replay_state cluster1 mirror image_11 started 2023-08-26T13:53:09.344 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster1 2023-08-26T13:53:09.344 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T13:53:09.345 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=image_11 2023-08-26T13:53:09.345 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local test_state=started 2023-08-26T13:53:09.345 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local status_result 2023-08-26T13:53:09.345 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local current_state=stopped 2023-08-26T13:53:09.345 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_11 2023-08-26T13:53:09.346 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local cluster_instance=cluster1 2023-08-26T13:53:09.346 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ grep -i state 2023-08-26T13:53:09.346 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ shift 2023-08-26T13:53:09.346 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local cluster=cluster1 2023-08-26T13:53:09.346 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local instance=cluster1 2023-08-26T13:53:09.346 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local loop_instance 2023-08-26T13:53:09.347 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-08-26T13:53:09.347 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ sleep 0 2023-08-26T13:53:09.355 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ '[' cluster1 '!=' cluster1 ']' 2023-08-26T13:53:09.356 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ seq 0 3 2023-08-26T13:53:09.356 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-08-26T13:53:09.356 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_11 2023-08-26T13:53:09.356 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local cluster=cluster1:0 2023-08-26T13:53:09.356 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ shift 2023-08-26T13:53:09.357 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local instance 2023-08-26T13:53:09.357 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ set_cluster_instance cluster1:0 cluster instance 2023-08-26T13:53:09.357 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local val=cluster1:0 2023-08-26T13:53:09.357 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local cluster_var_name=cluster 2023-08-26T13:53:09.357 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local instance_var_name=instance 2023-08-26T13:53:09.358 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ cluster=cluster1 2023-08-26T13:53:09.358 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ instance=0 2023-08-26T13:53:09.358 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ '[' 0 = cluster1:0 ']' 2023-08-26T13:53:09.358 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ eval cluster=cluster1 2023-08-26T13:53:09.358 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ cluster=cluster1 2023-08-26T13:53:09.359 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ eval instance=0 2023-08-26T13:53:09.359 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ instance=0 2023-08-26T13:53:09.359 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ daemon_asok_file cluster1:0 cluster1 2023-08-26T13:53:09.359 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ local local_cluster=cluster1:0 2023-08-26T13:53:09.359 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ local cluster=cluster1 2023-08-26T13:53:09.359 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ local instance 2023-08-26T13:53:09.360 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2023-08-26T13:53:09.360 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ local val=cluster1:0 2023-08-26T13:53:09.360 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ local cluster_var_name=local_cluster 2023-08-26T13:53:09.360 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ local instance_var_name=instance 2023-08-26T13:53:09.360 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ cluster=cluster1 2023-08-26T13:53:09.361 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ instance=0 2023-08-26T13:53:09.361 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ '[' 0 = cluster1:0 ']' 2023-08-26T13:53:09.361 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ eval local_cluster=cluster1 2023-08-26T13:53:09.361 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++++ local_cluster=cluster1 2023-08-26T13:53:09.361 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ eval instance=0 2023-08-26T13:53:09.361 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++++ instance=0 2023-08-26T13:53:09.362 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2023-08-26T13:53:09.369 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-08-26T13:53:09.370 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-08-26T13:53:09.370 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-08-26T13:53:09.370 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_11 2023-08-26T13:53:09.480 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:Invalid command: unused arguments: ['mirror/image_11'] 2023-08-26T13:53:09.480 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:rbd mirror status : get status for rbd mirror 2023-08-26T13:53:09.480 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:admin_socket: invalid command 2023-08-26T13:53:09.483 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-08-26T13:53:09.483 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_11 2023-08-26T13:53:09.483 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local cluster=cluster1:1 2023-08-26T13:53:09.484 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ shift 2023-08-26T13:53:09.484 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local instance 2023-08-26T13:53:09.484 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ set_cluster_instance cluster1:1 cluster instance 2023-08-26T13:53:09.484 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local val=cluster1:1 2023-08-26T13:53:09.484 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local cluster_var_name=cluster 2023-08-26T13:53:09.485 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local instance_var_name=instance 2023-08-26T13:53:09.485 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ cluster=cluster1 2023-08-26T13:53:09.485 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ instance=1 2023-08-26T13:53:09.485 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ '[' 1 = cluster1:1 ']' 2023-08-26T13:53:09.485 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ eval cluster=cluster1 2023-08-26T13:53:09.486 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ cluster=cluster1 2023-08-26T13:53:09.486 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ eval instance=1 2023-08-26T13:53:09.486 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ instance=1 2023-08-26T13:53:09.486 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ daemon_asok_file cluster1:1 cluster1 2023-08-26T13:53:09.486 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ local local_cluster=cluster1:1 2023-08-26T13:53:09.486 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ local cluster=cluster1 2023-08-26T13:53:09.487 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ local instance 2023-08-26T13:53:09.487 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2023-08-26T13:53:09.487 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ local val=cluster1:1 2023-08-26T13:53:09.487 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ local cluster_var_name=local_cluster 2023-08-26T13:53:09.487 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ local instance_var_name=instance 2023-08-26T13:53:09.488 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ cluster=cluster1 2023-08-26T13:53:09.488 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ instance=1 2023-08-26T13:53:09.488 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ '[' 1 = cluster1:1 ']' 2023-08-26T13:53:09.488 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ eval local_cluster=cluster1 2023-08-26T13:53:09.488 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++++ local_cluster=cluster1 2023-08-26T13:53:09.488 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ eval instance=1 2023-08-26T13:53:09.489 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++++ instance=1 2023-08-26T13:53:09.489 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2023-08-26T13:53:09.502 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-08-26T13:53:09.502 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-08-26T13:53:09.502 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-08-26T13:53:09.503 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_11 2023-08-26T13:53:09.618 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ return 0 2023-08-26T13:53:09.618 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ status_result=' "state": "Replaying"' 2023-08-26T13:53:09.619 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ echo ' "state": "Replaying"' 2023-08-26T13:53:09.619 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ grep -i Replaying 2023-08-26T13:53:09.619 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: "state": "Replaying" 2023-08-26T13:53:09.622 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ current_state=started 2023-08-26T13:53:09.622 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ test started = started 2023-08-26T13:53:09.622 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ return 0 2023-08-26T13:53:09.622 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_11 2023-08-26T13:53:09.623 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local local_cluster=cluster1 2023-08-26T13:53:09.623 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster2 2023-08-26T13:53:09.623 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T13:53:09.623 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=image_11 2023-08-26T13:53:09.623 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ '[' snapshot = journal ']' 2023-08-26T13:53:09.624 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ '[' snapshot = snapshot ']' 2023-08-26T13:53:09.624 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_11 2023-08-26T13:53:09.624 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local local_cluster=cluster1 2023-08-26T13:53:09.624 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster2 2023-08-26T13:53:09.625 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T13:53:09.625 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=image_11 2023-08-26T13:53:09.625 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ mkfname cluster2-mirror-image_11.status 2023-08-26T13:53:09.625 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ echo cluster2-mirror-image_11.status 2023-08-26T13:53:09.625 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ sed -e 's|[/ ]|_|g' 2023-08-26T13:53:09.626 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local status_log=/tmp/tmp.TfDn6BcQ3N/cluster2-mirror-image_11.status 2023-08-26T13:53:09.626 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ mkfname cluster1-mirror-image_11.status 2023-08-26T13:53:09.626 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ echo cluster1-mirror-image_11.status 2023-08-26T13:53:09.626 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ sed -e 's|[/ ]|_|g' 2023-08-26T13:53:09.627 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local local_status_log=/tmp/tmp.TfDn6BcQ3N/cluster1-mirror-image_11.status 2023-08-26T13:53:09.627 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ mirror_image_snapshot cluster2 mirror image_11 2023-08-26T13:53:09.628 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster2 2023-08-26T13:53:09.628 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T13:53:09.628 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=image_11 2023-08-26T13:53:09.628 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_11 2023-08-26T13:53:10.345 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:Snapshot ID: 139 2023-08-26T13:53:10.355 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_11 /tmp/tmp.TfDn6BcQ3N/cluster2-mirror-image_11.status 2023-08-26T13:53:10.355 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster2 2023-08-26T13:53:10.355 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T13:53:10.355 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=image_11 2023-08-26T13:53:10.356 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local log=/tmp/tmp.TfDn6BcQ3N/cluster2-mirror-image_11.status 2023-08-26T13:53:10.356 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_11 --format xml 2023-08-26T13:53:10.356 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-08-26T13:53:10.401 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ xmlstarlet sel -t -v //snapshot/id 2023-08-26T13:53:10.403 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local snapshot_id=139 2023-08-26T13:53:10.403 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ true 2023-08-26T13:53:10.403 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-08-26T13:53:10.403 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ sleep 0.2 2023-08-26T13:53:10.604 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_11 /tmp/tmp.TfDn6BcQ3N/cluster1-mirror-image_11.status 2023-08-26T13:53:10.604 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster1 2023-08-26T13:53:10.605 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T13:53:10.605 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=image_11 2023-08-26T13:53:10.605 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local log=/tmp/tmp.TfDn6BcQ3N/cluster1-mirror-image_11.status 2023-08-26T13:53:10.605 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_11 --format xml 2023-08-26T13:53:10.605 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-08-26T13:53:10.655 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-08-26T13:53:10.657 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local primary_snapshot_id=78 2023-08-26T13:53:10.657 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ test 139 = 78 2023-08-26T13:53:10.657 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-08-26T13:53:10.658 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ sleep 0.4 2023-08-26T13:53:11.058 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_11 /tmp/tmp.TfDn6BcQ3N/cluster1-mirror-image_11.status 2023-08-26T13:53:11.059 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster1 2023-08-26T13:53:11.059 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T13:53:11.059 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=image_11 2023-08-26T13:53:11.059 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local log=/tmp/tmp.TfDn6BcQ3N/cluster1-mirror-image_11.status 2023-08-26T13:53:11.059 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_11 --format xml 2023-08-26T13:53:11.060 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-08-26T13:53:11.108 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-08-26T13:53:11.110 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local primary_snapshot_id=78 2023-08-26T13:53:11.110 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ test 139 = 78 2023-08-26T13:53:11.111 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-08-26T13:53:11.111 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ sleep 0.8 2023-08-26T13:53:11.912 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_11 /tmp/tmp.TfDn6BcQ3N/cluster1-mirror-image_11.status 2023-08-26T13:53:11.912 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster1 2023-08-26T13:53:11.912 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T13:53:11.912 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=image_11 2023-08-26T13:53:11.913 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local log=/tmp/tmp.TfDn6BcQ3N/cluster1-mirror-image_11.status 2023-08-26T13:53:11.913 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_11 --format xml 2023-08-26T13:53:11.913 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-08-26T13:53:11.967 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ true 2023-08-26T13:53:11.967 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-08-26T13:53:11.968 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:-:1.1: Document is empty 2023-08-26T13:53:11.969 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr: 2023-08-26T13:53:11.969 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:^ 2023-08-26T13:53:11.970 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local primary_snapshot_id= 2023-08-26T13:53:11.970 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ test 139 = '' 2023-08-26T13:53:11.970 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-08-26T13:53:11.970 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ sleep 1.6 2023-08-26T13:53:13.571 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_11 /tmp/tmp.TfDn6BcQ3N/cluster1-mirror-image_11.status 2023-08-26T13:53:13.571 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster1 2023-08-26T13:53:13.571 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T13:53:13.571 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=image_11 2023-08-26T13:53:13.571 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local log=/tmp/tmp.TfDn6BcQ3N/cluster1-mirror-image_11.status 2023-08-26T13:53:13.572 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_11 --format xml 2023-08-26T13:53:13.572 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-08-26T13:53:13.622 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-08-26T13:53:13.625 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local primary_snapshot_id=139 2023-08-26T13:53:13.625 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ test 139 = 139 2023-08-26T13:53:13.626 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ return 0 2023-08-26T13:53:13.626 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ wait_for_snap_present cluster1 mirror image_11 snap 2023-08-26T13:53:13.626 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster1 2023-08-26T13:53:13.626 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T13:53:13.626 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=image_11 2023-08-26T13:53:13.627 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local snap_name=snap 2023-08-26T13:53:13.627 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local s 2023-08-26T13:53:13.627 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2023-08-26T13:53:13.627 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ sleep 1 2023-08-26T13:53:14.625 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rbd --cluster cluster1 info mirror/image_11@snap 2023-08-26T13:53:14.673 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:rbd image 'image_11': 2023-08-26T13:53:14.674 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: size 128 MiB in 32 objects 2023-08-26T13:53:14.674 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: order 22 (4 MiB objects) 2023-08-26T13:53:14.674 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: snapshot_count: 2 2023-08-26T13:53:14.674 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: id: 11d2345edfbf 2023-08-26T13:53:14.674 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: block_name_prefix: rbd_data.11d2345edfbf 2023-08-26T13:53:14.675 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: format: 2 2023-08-26T13:53:14.675 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: features: layering, exclusive-lock, non-primary 2023-08-26T13:53:14.675 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: op_features: 2023-08-26T13:53:14.675 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: flags: 2023-08-26T13:53:14.675 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: create_timestamp: Sat Aug 26 13:44:24 2023 2023-08-26T13:53:14.676 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: access_timestamp: Sat Aug 26 13:44:24 2023 2023-08-26T13:53:14.676 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: modify_timestamp: Sat Aug 26 13:44:24 2023 2023-08-26T13:53:14.676 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: protected: False 2023-08-26T13:53:14.676 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: mirroring state: enabled 2023-08-26T13:53:14.677 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: mirroring mode: snapshot 2023-08-26T13:53:14.677 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: mirroring global id: f979d3c9-8b28-4180-872b-79e338751061 2023-08-26T13:53:14.677 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: mirroring primary: false 2023-08-26T13:53:14.678 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ return 0 2023-08-26T13:53:14.679 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ compare_image_snaps mirror image_11 snap 2023-08-26T13:53:14.679 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T13:53:14.679 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=image_11 2023-08-26T13:53:14.680 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local snap_name=snap 2023-08-26T13:53:14.680 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local ret=0 2023-08-26T13:53:14.680 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local rmt_export=/tmp/tmp.TfDn6BcQ3N/cluster2-mirror-image_11.export 2023-08-26T13:53:14.680 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local loc_export=/tmp/tmp.TfDn6BcQ3N/cluster1-mirror-image_11.export 2023-08-26T13:53:14.681 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rm -f /tmp/tmp.TfDn6BcQ3N/cluster2-mirror-image_11.export /tmp/tmp.TfDn6BcQ3N/cluster1-mirror-image_11.export 2023-08-26T13:53:14.681 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rbd --cluster cluster2 -p mirror export image_11@snap /tmp/tmp.TfDn6BcQ3N/cluster2-mirror-image_11.export 2023-08-26T13:53:14.854 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-08-26T13:53:14.859 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rbd --cluster cluster1 -p mirror export image_11@snap /tmp/tmp.TfDn6BcQ3N/cluster1-mirror-image_11.export 2023-08-26T13:53:15.049 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-08-26T13:53:15.055 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ cmp /tmp/tmp.TfDn6BcQ3N/cluster2-mirror-image_11.export /tmp/tmp.TfDn6BcQ3N/cluster1-mirror-image_11.export 2023-08-26T13:53:15.111 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rm -f /tmp/tmp.TfDn6BcQ3N/cluster2-mirror-image_11.export /tmp/tmp.TfDn6BcQ3N/cluster1-mirror-image_11.export 2023-08-26T13:53:15.148 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ return 0 2023-08-26T13:53:15.148 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-08-26T13:53:15.148 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ image=image_12 2023-08-26T13:53:15.149 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ create_snap cluster2 mirror image_12 snap 2023-08-26T13:53:15.149 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster2 2023-08-26T13:53:15.149 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T13:53:15.149 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=image_12 2023-08-26T13:53:15.149 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local snap_name=snap 2023-08-26T13:53:15.150 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rbd --cluster cluster2 -p mirror snap create image_12@snap --debug-rbd=20 --debug-journaler=20 2023-08-26T13:53:16.355 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ wait_for_image_replay_started cluster1 mirror image_12 2023-08-26T13:53:16.355 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster1 2023-08-26T13:53:16.355 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T13:53:16.355 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=image_12 2023-08-26T13:53:16.356 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ wait_for_image_replay_state cluster1 mirror image_12 started 2023-08-26T13:53:16.356 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster1 2023-08-26T13:53:16.356 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T13:53:16.356 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=image_12 2023-08-26T13:53:16.356 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local state=started 2023-08-26T13:53:16.357 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local s 2023-08-26T13:53:16.357 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-08-26T13:53:16.357 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ sleep 1 2023-08-26T13:53:17.358 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ test_image_replay_state cluster1 mirror image_12 started 2023-08-26T13:53:17.358 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster1 2023-08-26T13:53:17.359 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T13:53:17.359 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=image_12 2023-08-26T13:53:17.359 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local test_state=started 2023-08-26T13:53:17.359 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local status_result 2023-08-26T13:53:17.359 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local current_state=stopped 2023-08-26T13:53:17.360 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_12 2023-08-26T13:53:17.360 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ grep -i state 2023-08-26T13:53:17.360 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local cluster_instance=cluster1 2023-08-26T13:53:17.360 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ shift 2023-08-26T13:53:17.361 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local cluster=cluster1 2023-08-26T13:53:17.361 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local instance=cluster1 2023-08-26T13:53:17.361 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local loop_instance 2023-08-26T13:53:17.361 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-08-26T13:53:17.362 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ sleep 0 2023-08-26T13:53:17.362 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ '[' cluster1 '!=' cluster1 ']' 2023-08-26T13:53:17.362 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ seq 0 3 2023-08-26T13:53:17.365 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-08-26T13:53:17.365 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_12 2023-08-26T13:53:17.366 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local cluster=cluster1:0 2023-08-26T13:53:17.366 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ shift 2023-08-26T13:53:17.366 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local instance 2023-08-26T13:53:17.366 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ set_cluster_instance cluster1:0 cluster instance 2023-08-26T13:53:17.366 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local val=cluster1:0 2023-08-26T13:53:17.366 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local cluster_var_name=cluster 2023-08-26T13:53:17.367 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local instance_var_name=instance 2023-08-26T13:53:17.367 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ cluster=cluster1 2023-08-26T13:53:17.367 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ instance=0 2023-08-26T13:53:17.367 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ '[' 0 = cluster1:0 ']' 2023-08-26T13:53:17.367 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ eval cluster=cluster1 2023-08-26T13:53:17.368 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ cluster=cluster1 2023-08-26T13:53:17.368 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ eval instance=0 2023-08-26T13:53:17.368 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ instance=0 2023-08-26T13:53:17.368 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ daemon_asok_file cluster1:0 cluster1 2023-08-26T13:53:17.368 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ local local_cluster=cluster1:0 2023-08-26T13:53:17.369 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ local cluster=cluster1 2023-08-26T13:53:17.369 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ local instance 2023-08-26T13:53:17.369 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2023-08-26T13:53:17.369 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ local val=cluster1:0 2023-08-26T13:53:17.369 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ local cluster_var_name=local_cluster 2023-08-26T13:53:17.369 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ local instance_var_name=instance 2023-08-26T13:53:17.370 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ cluster=cluster1 2023-08-26T13:53:17.370 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ instance=0 2023-08-26T13:53:17.370 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ '[' 0 = cluster1:0 ']' 2023-08-26T13:53:17.370 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ eval local_cluster=cluster1 2023-08-26T13:53:17.370 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++++ local_cluster=cluster1 2023-08-26T13:53:17.370 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ eval instance=0 2023-08-26T13:53:17.371 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++++ instance=0 2023-08-26T13:53:17.371 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2023-08-26T13:53:17.383 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-08-26T13:53:17.383 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-08-26T13:53:17.383 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-08-26T13:53:17.383 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_12 2023-08-26T13:53:17.501 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ return 0 2023-08-26T13:53:17.502 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ status_result=' "state": "Replaying"' 2023-08-26T13:53:17.502 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ echo ' "state": "Replaying"' 2023-08-26T13:53:17.503 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ grep -i Replaying 2023-08-26T13:53:17.503 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: "state": "Replaying" 2023-08-26T13:53:17.505 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ current_state=started 2023-08-26T13:53:17.505 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ test started = started 2023-08-26T13:53:17.505 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ return 0 2023-08-26T13:53:17.506 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_12 2023-08-26T13:53:17.506 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local local_cluster=cluster1 2023-08-26T13:53:17.506 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster2 2023-08-26T13:53:17.506 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T13:53:17.506 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=image_12 2023-08-26T13:53:17.507 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ '[' snapshot = journal ']' 2023-08-26T13:53:17.507 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ '[' snapshot = snapshot ']' 2023-08-26T13:53:17.507 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_12 2023-08-26T13:53:17.507 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local local_cluster=cluster1 2023-08-26T13:53:17.507 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster2 2023-08-26T13:53:17.507 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T13:53:17.508 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=image_12 2023-08-26T13:53:17.508 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ mkfname cluster2-mirror-image_12.status 2023-08-26T13:53:17.508 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ echo cluster2-mirror-image_12.status 2023-08-26T13:53:17.508 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ sed -e 's|[/ ]|_|g' 2023-08-26T13:53:17.508 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local status_log=/tmp/tmp.TfDn6BcQ3N/cluster2-mirror-image_12.status 2023-08-26T13:53:17.509 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ mkfname cluster1-mirror-image_12.status 2023-08-26T13:53:17.509 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ echo cluster1-mirror-image_12.status 2023-08-26T13:53:17.509 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ sed -e 's|[/ ]|_|g' 2023-08-26T13:53:17.510 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local local_status_log=/tmp/tmp.TfDn6BcQ3N/cluster1-mirror-image_12.status 2023-08-26T13:53:17.510 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ mirror_image_snapshot cluster2 mirror image_12 2023-08-26T13:53:17.510 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster2 2023-08-26T13:53:17.511 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T13:53:17.511 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=image_12 2023-08-26T13:53:17.511 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_12 2023-08-26T13:53:18.357 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:Snapshot ID: 141 2023-08-26T13:53:18.365 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_12 /tmp/tmp.TfDn6BcQ3N/cluster2-mirror-image_12.status 2023-08-26T13:53:18.366 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster2 2023-08-26T13:53:18.366 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T13:53:18.366 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=image_12 2023-08-26T13:53:18.366 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local log=/tmp/tmp.TfDn6BcQ3N/cluster2-mirror-image_12.status 2023-08-26T13:53:18.367 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_12 --format xml 2023-08-26T13:53:18.367 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-08-26T13:53:18.411 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ xmlstarlet sel -t -v //snapshot/id 2023-08-26T13:53:18.413 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local snapshot_id=141 2023-08-26T13:53:18.413 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ true 2023-08-26T13:53:18.413 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-08-26T13:53:18.413 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ sleep 0.2 2023-08-26T13:53:18.614 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_12 /tmp/tmp.TfDn6BcQ3N/cluster1-mirror-image_12.status 2023-08-26T13:53:18.615 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster1 2023-08-26T13:53:18.615 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T13:53:18.615 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=image_12 2023-08-26T13:53:18.615 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local log=/tmp/tmp.TfDn6BcQ3N/cluster1-mirror-image_12.status 2023-08-26T13:53:18.615 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_12 --format xml 2023-08-26T13:53:18.616 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-08-26T13:53:18.663 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ true 2023-08-26T13:53:18.663 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-08-26T13:53:18.664 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:-:1.1: Document is empty 2023-08-26T13:53:18.664 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr: 2023-08-26T13:53:18.664 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:^ 2023-08-26T13:53:18.665 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local primary_snapshot_id= 2023-08-26T13:53:18.665 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ test 141 = '' 2023-08-26T13:53:18.665 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-08-26T13:53:18.665 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ sleep 0.4 2023-08-26T13:53:19.067 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_12 /tmp/tmp.TfDn6BcQ3N/cluster1-mirror-image_12.status 2023-08-26T13:53:19.067 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster1 2023-08-26T13:53:19.067 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T13:53:19.068 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=image_12 2023-08-26T13:53:19.068 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local log=/tmp/tmp.TfDn6BcQ3N/cluster1-mirror-image_12.status 2023-08-26T13:53:19.068 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_12 --format xml 2023-08-26T13:53:19.069 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-08-26T13:53:19.115 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ true 2023-08-26T13:53:19.115 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-08-26T13:53:19.116 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:-:1.1: Document is empty 2023-08-26T13:53:19.116 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr: 2023-08-26T13:53:19.117 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:^ 2023-08-26T13:53:19.117 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local primary_snapshot_id= 2023-08-26T13:53:19.118 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ test 141 = '' 2023-08-26T13:53:19.118 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-08-26T13:53:19.118 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ sleep 0.8 2023-08-26T13:53:19.918 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_12 /tmp/tmp.TfDn6BcQ3N/cluster1-mirror-image_12.status 2023-08-26T13:53:19.918 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster1 2023-08-26T13:53:19.919 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T13:53:19.919 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=image_12 2023-08-26T13:53:19.919 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local log=/tmp/tmp.TfDn6BcQ3N/cluster1-mirror-image_12.status 2023-08-26T13:53:19.919 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_12 --format xml 2023-08-26T13:53:19.920 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-08-26T13:53:19.963 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-08-26T13:53:19.965 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local primary_snapshot_id=141 2023-08-26T13:53:19.965 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ test 141 = 141 2023-08-26T13:53:19.966 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ return 0 2023-08-26T13:53:19.966 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ wait_for_snap_present cluster1 mirror image_12 snap 2023-08-26T13:53:19.966 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster1 2023-08-26T13:53:19.966 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T13:53:19.967 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=image_12 2023-08-26T13:53:19.967 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local snap_name=snap 2023-08-26T13:53:19.967 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local s 2023-08-26T13:53:19.967 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2023-08-26T13:53:19.967 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ sleep 1 2023-08-26T13:53:20.966 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rbd --cluster cluster1 info mirror/image_12@snap 2023-08-26T13:53:21.011 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:rbd image 'image_12': 2023-08-26T13:53:21.012 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: size 128 MiB in 32 objects 2023-08-26T13:53:21.012 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: order 22 (4 MiB objects) 2023-08-26T13:53:21.012 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: snapshot_count: 2 2023-08-26T13:53:21.013 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: id: 12b65b996a94 2023-08-26T13:53:21.013 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: block_name_prefix: rbd_data.12b65b996a94 2023-08-26T13:53:21.013 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: format: 2 2023-08-26T13:53:21.013 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: features: layering, exclusive-lock, non-primary 2023-08-26T13:53:21.014 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: op_features: 2023-08-26T13:53:21.014 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: flags: 2023-08-26T13:53:21.014 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: create_timestamp: Sat Aug 26 13:44:34 2023 2023-08-26T13:53:21.015 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: access_timestamp: Sat Aug 26 13:44:34 2023 2023-08-26T13:53:21.015 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: modify_timestamp: Sat Aug 26 13:44:34 2023 2023-08-26T13:53:21.015 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: protected: False 2023-08-26T13:53:21.015 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: mirroring state: enabled 2023-08-26T13:53:21.016 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: mirroring mode: snapshot 2023-08-26T13:53:21.016 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: mirroring global id: b189e689-d710-4185-9857-d4ed9dbb9df5 2023-08-26T13:53:21.016 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: mirroring primary: false 2023-08-26T13:53:21.017 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ return 0 2023-08-26T13:53:21.018 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ compare_image_snaps mirror image_12 snap 2023-08-26T13:53:21.018 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T13:53:21.018 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=image_12 2023-08-26T13:53:21.018 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local snap_name=snap 2023-08-26T13:53:21.019 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local ret=0 2023-08-26T13:53:21.019 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local rmt_export=/tmp/tmp.TfDn6BcQ3N/cluster2-mirror-image_12.export 2023-08-26T13:53:21.020 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local loc_export=/tmp/tmp.TfDn6BcQ3N/cluster1-mirror-image_12.export 2023-08-26T13:53:21.020 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rm -f /tmp/tmp.TfDn6BcQ3N/cluster2-mirror-image_12.export /tmp/tmp.TfDn6BcQ3N/cluster1-mirror-image_12.export 2023-08-26T13:53:21.020 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rbd --cluster cluster2 -p mirror export image_12@snap /tmp/tmp.TfDn6BcQ3N/cluster2-mirror-image_12.export 2023-08-26T13:53:21.199 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-08-26T13:53:21.206 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rbd --cluster cluster1 -p mirror export image_12@snap /tmp/tmp.TfDn6BcQ3N/cluster1-mirror-image_12.export 2023-08-26T13:53:21.419 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-08-26T13:53:21.425 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ cmp /tmp/tmp.TfDn6BcQ3N/cluster2-mirror-image_12.export /tmp/tmp.TfDn6BcQ3N/cluster1-mirror-image_12.export 2023-08-26T13:53:21.480 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rm -f /tmp/tmp.TfDn6BcQ3N/cluster2-mirror-image_12.export /tmp/tmp.TfDn6BcQ3N/cluster1-mirror-image_12.export 2023-08-26T13:53:21.518 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ return 0 2023-08-26T13:53:21.518 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-08-26T13:53:21.519 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ image=image_13 2023-08-26T13:53:21.519 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ create_snap cluster2 mirror image_13 snap 2023-08-26T13:53:21.519 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster2 2023-08-26T13:53:21.519 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T13:53:21.519 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=image_13 2023-08-26T13:53:21.520 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local snap_name=snap 2023-08-26T13:53:21.520 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rbd --cluster cluster2 -p mirror snap create image_13@snap --debug-rbd=20 --debug-journaler=20 2023-08-26T13:53:23.375 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ wait_for_image_replay_started cluster1 mirror image_13 2023-08-26T13:53:23.375 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster1 2023-08-26T13:53:23.375 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T13:53:23.375 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=image_13 2023-08-26T13:53:23.376 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ wait_for_image_replay_state cluster1 mirror image_13 started 2023-08-26T13:53:23.376 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster1 2023-08-26T13:53:23.376 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T13:53:23.376 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=image_13 2023-08-26T13:53:23.376 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local state=started 2023-08-26T13:53:23.377 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local s 2023-08-26T13:53:23.377 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-08-26T13:53:23.377 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ sleep 1 2023-08-26T13:53:24.377 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ test_image_replay_state cluster1 mirror image_13 started 2023-08-26T13:53:24.377 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster1 2023-08-26T13:53:24.377 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T13:53:24.377 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=image_13 2023-08-26T13:53:24.377 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local test_state=started 2023-08-26T13:53:24.378 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local status_result 2023-08-26T13:53:24.378 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local current_state=stopped 2023-08-26T13:53:24.378 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_13 2023-08-26T13:53:24.378 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local cluster_instance=cluster1 2023-08-26T13:53:24.378 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ shift 2023-08-26T13:53:24.379 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ grep -i state 2023-08-26T13:53:24.379 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local cluster=cluster1 2023-08-26T13:53:24.379 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local instance=cluster1 2023-08-26T13:53:24.379 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local loop_instance 2023-08-26T13:53:24.379 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-08-26T13:53:24.380 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ sleep 0 2023-08-26T13:53:24.382 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ '[' cluster1 '!=' cluster1 ']' 2023-08-26T13:53:24.382 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ seq 0 3 2023-08-26T13:53:24.382 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-08-26T13:53:24.382 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_13 2023-08-26T13:53:24.382 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local cluster=cluster1:0 2023-08-26T13:53:24.383 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ shift 2023-08-26T13:53:24.383 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local instance 2023-08-26T13:53:24.383 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ set_cluster_instance cluster1:0 cluster instance 2023-08-26T13:53:24.383 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local val=cluster1:0 2023-08-26T13:53:24.383 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local cluster_var_name=cluster 2023-08-26T13:53:24.384 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local instance_var_name=instance 2023-08-26T13:53:24.384 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ cluster=cluster1 2023-08-26T13:53:24.384 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ instance=0 2023-08-26T13:53:24.384 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ '[' 0 = cluster1:0 ']' 2023-08-26T13:53:24.384 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ eval cluster=cluster1 2023-08-26T13:53:24.385 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ cluster=cluster1 2023-08-26T13:53:24.385 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ eval instance=0 2023-08-26T13:53:24.385 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ instance=0 2023-08-26T13:53:24.385 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ daemon_asok_file cluster1:0 cluster1 2023-08-26T13:53:24.385 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ local local_cluster=cluster1:0 2023-08-26T13:53:24.387 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ local cluster=cluster1 2023-08-26T13:53:24.387 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ local instance 2023-08-26T13:53:24.387 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2023-08-26T13:53:24.388 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ local val=cluster1:0 2023-08-26T13:53:24.388 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ local cluster_var_name=local_cluster 2023-08-26T13:53:24.388 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ local instance_var_name=instance 2023-08-26T13:53:24.388 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ cluster=cluster1 2023-08-26T13:53:24.388 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ instance=0 2023-08-26T13:53:24.389 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ '[' 0 = cluster1:0 ']' 2023-08-26T13:53:24.389 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ eval local_cluster=cluster1 2023-08-26T13:53:24.389 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++++ local_cluster=cluster1 2023-08-26T13:53:24.389 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ eval instance=0 2023-08-26T13:53:24.389 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++++ instance=0 2023-08-26T13:53:24.390 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2023-08-26T13:53:24.403 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-08-26T13:53:24.403 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-08-26T13:53:24.403 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-08-26T13:53:24.403 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_13 2023-08-26T13:53:24.513 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:Invalid command: unused arguments: ['mirror/image_13'] 2023-08-26T13:53:24.513 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:rbd mirror status : get status for rbd mirror 2023-08-26T13:53:24.513 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:admin_socket: invalid command 2023-08-26T13:53:24.517 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-08-26T13:53:24.517 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_13 2023-08-26T13:53:24.517 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local cluster=cluster1:1 2023-08-26T13:53:24.518 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ shift 2023-08-26T13:53:24.518 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local instance 2023-08-26T13:53:24.518 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ set_cluster_instance cluster1:1 cluster instance 2023-08-26T13:53:24.518 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local val=cluster1:1 2023-08-26T13:53:24.518 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local cluster_var_name=cluster 2023-08-26T13:53:24.519 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local instance_var_name=instance 2023-08-26T13:53:24.519 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ cluster=cluster1 2023-08-26T13:53:24.519 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ instance=1 2023-08-26T13:53:24.519 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ '[' 1 = cluster1:1 ']' 2023-08-26T13:53:24.519 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ eval cluster=cluster1 2023-08-26T13:53:24.520 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ cluster=cluster1 2023-08-26T13:53:24.520 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ eval instance=1 2023-08-26T13:53:24.520 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ instance=1 2023-08-26T13:53:24.520 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ daemon_asok_file cluster1:1 cluster1 2023-08-26T13:53:24.520 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ local local_cluster=cluster1:1 2023-08-26T13:53:24.521 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ local cluster=cluster1 2023-08-26T13:53:24.521 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ local instance 2023-08-26T13:53:24.521 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2023-08-26T13:53:24.521 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ local val=cluster1:1 2023-08-26T13:53:24.521 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ local cluster_var_name=local_cluster 2023-08-26T13:53:24.522 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ local instance_var_name=instance 2023-08-26T13:53:24.522 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ cluster=cluster1 2023-08-26T13:53:24.522 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ instance=1 2023-08-26T13:53:24.522 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ '[' 1 = cluster1:1 ']' 2023-08-26T13:53:24.522 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ eval local_cluster=cluster1 2023-08-26T13:53:24.523 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++++ local_cluster=cluster1 2023-08-26T13:53:24.523 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ eval instance=1 2023-08-26T13:53:24.523 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++++ instance=1 2023-08-26T13:53:24.523 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2023-08-26T13:53:24.535 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-08-26T13:53:24.535 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-08-26T13:53:24.535 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-08-26T13:53:24.536 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_13 2023-08-26T13:53:24.641 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:Invalid command: unused arguments: ['mirror/image_13'] 2023-08-26T13:53:24.641 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:rbd mirror status : get status for rbd mirror 2023-08-26T13:53:24.641 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:admin_socket: invalid command 2023-08-26T13:53:24.644 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-08-26T13:53:24.645 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/image_13 2023-08-26T13:53:24.645 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local cluster=cluster1:2 2023-08-26T13:53:24.645 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ shift 2023-08-26T13:53:24.645 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local instance 2023-08-26T13:53:24.645 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ set_cluster_instance cluster1:2 cluster instance 2023-08-26T13:53:24.646 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local val=cluster1:2 2023-08-26T13:53:24.646 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local cluster_var_name=cluster 2023-08-26T13:53:24.646 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local instance_var_name=instance 2023-08-26T13:53:24.646 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ cluster=cluster1 2023-08-26T13:53:24.646 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ instance=2 2023-08-26T13:53:24.647 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ '[' 2 = cluster1:2 ']' 2023-08-26T13:53:24.647 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ eval cluster=cluster1 2023-08-26T13:53:24.647 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ cluster=cluster1 2023-08-26T13:53:24.647 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ eval instance=2 2023-08-26T13:53:24.648 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ instance=2 2023-08-26T13:53:24.648 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ daemon_asok_file cluster1:2 cluster1 2023-08-26T13:53:24.648 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ local local_cluster=cluster1:2 2023-08-26T13:53:24.648 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ local cluster=cluster1 2023-08-26T13:53:24.648 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ local instance 2023-08-26T13:53:24.648 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2023-08-26T13:53:24.649 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ local val=cluster1:2 2023-08-26T13:53:24.649 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ local cluster_var_name=local_cluster 2023-08-26T13:53:24.649 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ local instance_var_name=instance 2023-08-26T13:53:24.649 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ cluster=cluster1 2023-08-26T13:53:24.649 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ instance=2 2023-08-26T13:53:24.650 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ '[' 2 = cluster1:2 ']' 2023-08-26T13:53:24.650 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ eval local_cluster=cluster1 2023-08-26T13:53:24.650 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++++ local_cluster=cluster1 2023-08-26T13:53:24.650 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ eval instance=2 2023-08-26T13:53:24.650 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++++ instance=2 2023-08-26T13:53:24.650 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2023-08-26T13:53:24.663 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-08-26T13:53:24.664 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-08-26T13:53:24.664 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-08-26T13:53:24.664 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/image_13 2023-08-26T13:53:24.777 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ return 0 2023-08-26T13:53:24.779 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ status_result=' "state": "Replaying"' 2023-08-26T13:53:24.779 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ echo ' "state": "Replaying"' 2023-08-26T13:53:24.779 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ grep -i Replaying 2023-08-26T13:53:24.779 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: "state": "Replaying" 2023-08-26T13:53:24.786 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ current_state=started 2023-08-26T13:53:24.787 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ test started = started 2023-08-26T13:53:24.787 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ return 0 2023-08-26T13:53:24.787 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_13 2023-08-26T13:53:24.787 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local local_cluster=cluster1 2023-08-26T13:53:24.787 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster2 2023-08-26T13:53:24.788 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T13:53:24.788 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=image_13 2023-08-26T13:53:24.788 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ '[' snapshot = journal ']' 2023-08-26T13:53:24.788 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ '[' snapshot = snapshot ']' 2023-08-26T13:53:24.788 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_13 2023-08-26T13:53:24.789 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local local_cluster=cluster1 2023-08-26T13:53:24.789 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster2 2023-08-26T13:53:24.789 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T13:53:24.789 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=image_13 2023-08-26T13:53:24.789 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ mkfname cluster2-mirror-image_13.status 2023-08-26T13:53:24.789 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ echo cluster2-mirror-image_13.status 2023-08-26T13:53:24.790 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ sed -e 's|[/ ]|_|g' 2023-08-26T13:53:24.790 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local status_log=/tmp/tmp.TfDn6BcQ3N/cluster2-mirror-image_13.status 2023-08-26T13:53:24.790 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ mkfname cluster1-mirror-image_13.status 2023-08-26T13:53:24.790 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ echo cluster1-mirror-image_13.status 2023-08-26T13:53:24.790 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ sed -e 's|[/ ]|_|g' 2023-08-26T13:53:24.791 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local local_status_log=/tmp/tmp.TfDn6BcQ3N/cluster1-mirror-image_13.status 2023-08-26T13:53:24.791 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ mirror_image_snapshot cluster2 mirror image_13 2023-08-26T13:53:24.791 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster2 2023-08-26T13:53:24.791 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T13:53:24.791 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=image_13 2023-08-26T13:53:24.792 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_13 2023-08-26T13:53:25.380 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:Snapshot ID: 143 2023-08-26T13:53:25.389 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_13 /tmp/tmp.TfDn6BcQ3N/cluster2-mirror-image_13.status 2023-08-26T13:53:25.390 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster2 2023-08-26T13:53:25.390 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T13:53:25.390 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=image_13 2023-08-26T13:53:25.390 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local log=/tmp/tmp.TfDn6BcQ3N/cluster2-mirror-image_13.status 2023-08-26T13:53:25.390 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_13 --format xml 2023-08-26T13:53:25.391 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-08-26T13:53:25.433 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ xmlstarlet sel -t -v //snapshot/id 2023-08-26T13:53:25.435 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local snapshot_id=143 2023-08-26T13:53:25.435 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ true 2023-08-26T13:53:25.436 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-08-26T13:53:25.436 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ sleep 0.2 2023-08-26T13:53:25.637 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_13 /tmp/tmp.TfDn6BcQ3N/cluster1-mirror-image_13.status 2023-08-26T13:53:25.637 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster1 2023-08-26T13:53:25.637 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T13:53:25.637 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=image_13 2023-08-26T13:53:25.637 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local log=/tmp/tmp.TfDn6BcQ3N/cluster1-mirror-image_13.status 2023-08-26T13:53:25.638 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_13 --format xml 2023-08-26T13:53:25.639 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-08-26T13:53:25.692 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-08-26T13:53:25.694 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local primary_snapshot_id=80 2023-08-26T13:53:25.694 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ test 143 = 80 2023-08-26T13:53:25.694 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-08-26T13:53:25.694 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ sleep 0.4 2023-08-26T13:53:26.095 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_13 /tmp/tmp.TfDn6BcQ3N/cluster1-mirror-image_13.status 2023-08-26T13:53:26.096 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster1 2023-08-26T13:53:26.096 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T13:53:26.096 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=image_13 2023-08-26T13:53:26.096 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local log=/tmp/tmp.TfDn6BcQ3N/cluster1-mirror-image_13.status 2023-08-26T13:53:26.097 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_13 --format xml 2023-08-26T13:53:26.097 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-08-26T13:53:26.144 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-08-26T13:53:26.146 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local primary_snapshot_id=80 2023-08-26T13:53:26.147 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ test 143 = 80 2023-08-26T13:53:26.147 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-08-26T13:53:26.147 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ sleep 0.8 2023-08-26T13:53:26.948 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_13 /tmp/tmp.TfDn6BcQ3N/cluster1-mirror-image_13.status 2023-08-26T13:53:26.948 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster1 2023-08-26T13:53:26.948 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T13:53:26.949 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=image_13 2023-08-26T13:53:26.949 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local log=/tmp/tmp.TfDn6BcQ3N/cluster1-mirror-image_13.status 2023-08-26T13:53:26.949 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_13 --format xml 2023-08-26T13:53:26.949 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-08-26T13:53:27.007 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ true 2023-08-26T13:53:27.008 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-08-26T13:53:27.009 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:-:1.1: Document is empty 2023-08-26T13:53:27.009 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr: 2023-08-26T13:53:27.009 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:^ 2023-08-26T13:53:27.010 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local primary_snapshot_id= 2023-08-26T13:53:27.010 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ test 143 = '' 2023-08-26T13:53:27.011 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-08-26T13:53:27.011 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ sleep 1.6 2023-08-26T13:53:28.611 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_13 /tmp/tmp.TfDn6BcQ3N/cluster1-mirror-image_13.status 2023-08-26T13:53:28.611 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster1 2023-08-26T13:53:28.612 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T13:53:28.612 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=image_13 2023-08-26T13:53:28.612 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local log=/tmp/tmp.TfDn6BcQ3N/cluster1-mirror-image_13.status 2023-08-26T13:53:28.612 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_13 --format xml 2023-08-26T13:53:28.612 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-08-26T13:53:28.663 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-08-26T13:53:28.665 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local primary_snapshot_id=143 2023-08-26T13:53:28.665 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ test 143 = 143 2023-08-26T13:53:28.666 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ return 0 2023-08-26T13:53:28.666 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ wait_for_snap_present cluster1 mirror image_13 snap 2023-08-26T13:53:28.666 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster1 2023-08-26T13:53:28.666 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T13:53:28.666 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=image_13 2023-08-26T13:53:28.666 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local snap_name=snap 2023-08-26T13:53:28.667 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local s 2023-08-26T13:53:28.667 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2023-08-26T13:53:28.667 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ sleep 1 2023-08-26T13:53:29.666 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rbd --cluster cluster1 info mirror/image_13@snap 2023-08-26T13:53:29.713 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:rbd image 'image_13': 2023-08-26T13:53:29.713 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: size 128 MiB in 32 objects 2023-08-26T13:53:29.713 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: order 22 (4 MiB objects) 2023-08-26T13:53:29.714 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: snapshot_count: 2 2023-08-26T13:53:29.714 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: id: 10b75798d681 2023-08-26T13:53:29.714 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: block_name_prefix: rbd_data.10b75798d681 2023-08-26T13:53:29.714 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: format: 2 2023-08-26T13:53:29.714 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: features: layering, exclusive-lock, non-primary 2023-08-26T13:53:29.715 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: op_features: 2023-08-26T13:53:29.715 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: flags: 2023-08-26T13:53:29.715 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: create_timestamp: Sat Aug 26 13:44:46 2023 2023-08-26T13:53:29.715 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: access_timestamp: Sat Aug 26 13:44:46 2023 2023-08-26T13:53:29.715 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: modify_timestamp: Sat Aug 26 13:44:46 2023 2023-08-26T13:53:29.716 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: protected: False 2023-08-26T13:53:29.716 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: mirroring state: enabled 2023-08-26T13:53:29.717 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: mirroring mode: snapshot 2023-08-26T13:53:29.717 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: mirroring global id: 3f172be8-5137-4cda-906a-0c674ef9c674 2023-08-26T13:53:29.717 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: mirroring primary: false 2023-08-26T13:53:29.718 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ return 0 2023-08-26T13:53:29.718 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ compare_image_snaps mirror image_13 snap 2023-08-26T13:53:29.718 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T13:53:29.719 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=image_13 2023-08-26T13:53:29.719 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local snap_name=snap 2023-08-26T13:53:29.719 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local ret=0 2023-08-26T13:53:29.719 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local rmt_export=/tmp/tmp.TfDn6BcQ3N/cluster2-mirror-image_13.export 2023-08-26T13:53:29.719 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local loc_export=/tmp/tmp.TfDn6BcQ3N/cluster1-mirror-image_13.export 2023-08-26T13:53:29.720 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rm -f /tmp/tmp.TfDn6BcQ3N/cluster2-mirror-image_13.export /tmp/tmp.TfDn6BcQ3N/cluster1-mirror-image_13.export 2023-08-26T13:53:29.720 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rbd --cluster cluster2 -p mirror export image_13@snap /tmp/tmp.TfDn6BcQ3N/cluster2-mirror-image_13.export 2023-08-26T13:53:29.900 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-08-26T13:53:29.906 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rbd --cluster cluster1 -p mirror export image_13@snap /tmp/tmp.TfDn6BcQ3N/cluster1-mirror-image_13.export 2023-08-26T13:53:30.122 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-08-26T13:53:30.128 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ cmp /tmp/tmp.TfDn6BcQ3N/cluster2-mirror-image_13.export /tmp/tmp.TfDn6BcQ3N/cluster1-mirror-image_13.export 2023-08-26T13:53:30.184 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rm -f /tmp/tmp.TfDn6BcQ3N/cluster2-mirror-image_13.export /tmp/tmp.TfDn6BcQ3N/cluster1-mirror-image_13.export 2023-08-26T13:53:30.221 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ return 0 2023-08-26T13:53:30.221 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-08-26T13:53:30.221 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ image=image_14 2023-08-26T13:53:30.221 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ create_snap cluster2 mirror image_14 snap 2023-08-26T13:53:30.222 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster2 2023-08-26T13:53:30.222 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T13:53:30.222 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=image_14 2023-08-26T13:53:30.222 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local snap_name=snap 2023-08-26T13:53:30.222 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rbd --cluster cluster2 -p mirror snap create image_14@snap --debug-rbd=20 --debug-journaler=20 2023-08-26T13:53:31.414 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ wait_for_image_replay_started cluster1 mirror image_14 2023-08-26T13:53:31.415 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster1 2023-08-26T13:53:31.415 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T13:53:31.415 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=image_14 2023-08-26T13:53:31.416 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ wait_for_image_replay_state cluster1 mirror image_14 started 2023-08-26T13:53:31.416 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster1 2023-08-26T13:53:31.416 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T13:53:31.417 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=image_14 2023-08-26T13:53:31.417 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local state=started 2023-08-26T13:53:31.417 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local s 2023-08-26T13:53:31.417 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-08-26T13:53:31.418 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ sleep 1 2023-08-26T13:53:32.415 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ test_image_replay_state cluster1 mirror image_14 started 2023-08-26T13:53:32.415 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster1 2023-08-26T13:53:32.416 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T13:53:32.416 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=image_14 2023-08-26T13:53:32.416 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local test_state=started 2023-08-26T13:53:32.416 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local status_result 2023-08-26T13:53:32.416 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local current_state=stopped 2023-08-26T13:53:32.418 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_14 2023-08-26T13:53:32.418 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ grep -i state 2023-08-26T13:53:32.418 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local cluster_instance=cluster1 2023-08-26T13:53:32.418 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ shift 2023-08-26T13:53:32.418 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local cluster=cluster1 2023-08-26T13:53:32.419 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local instance=cluster1 2023-08-26T13:53:32.419 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local loop_instance 2023-08-26T13:53:32.419 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-08-26T13:53:32.419 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ sleep 0 2023-08-26T13:53:32.419 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ '[' cluster1 '!=' cluster1 ']' 2023-08-26T13:53:32.420 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ seq 0 3 2023-08-26T13:53:32.422 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-08-26T13:53:32.423 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_14 2023-08-26T13:53:32.423 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local cluster=cluster1:0 2023-08-26T13:53:32.423 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ shift 2023-08-26T13:53:32.423 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local instance 2023-08-26T13:53:32.423 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ set_cluster_instance cluster1:0 cluster instance 2023-08-26T13:53:32.424 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local val=cluster1:0 2023-08-26T13:53:32.424 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local cluster_var_name=cluster 2023-08-26T13:53:32.424 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local instance_var_name=instance 2023-08-26T13:53:32.424 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ cluster=cluster1 2023-08-26T13:53:32.424 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ instance=0 2023-08-26T13:53:32.425 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ '[' 0 = cluster1:0 ']' 2023-08-26T13:53:32.425 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ eval cluster=cluster1 2023-08-26T13:53:32.425 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ cluster=cluster1 2023-08-26T13:53:32.425 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ eval instance=0 2023-08-26T13:53:32.425 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ instance=0 2023-08-26T13:53:32.425 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ daemon_asok_file cluster1:0 cluster1 2023-08-26T13:53:32.426 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ local local_cluster=cluster1:0 2023-08-26T13:53:32.426 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ local cluster=cluster1 2023-08-26T13:53:32.426 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ local instance 2023-08-26T13:53:32.426 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2023-08-26T13:53:32.426 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ local val=cluster1:0 2023-08-26T13:53:32.427 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ local cluster_var_name=local_cluster 2023-08-26T13:53:32.427 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ local instance_var_name=instance 2023-08-26T13:53:32.427 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ cluster=cluster1 2023-08-26T13:53:32.427 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ instance=0 2023-08-26T13:53:32.427 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ '[' 0 = cluster1:0 ']' 2023-08-26T13:53:32.428 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ eval local_cluster=cluster1 2023-08-26T13:53:32.428 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++++ local_cluster=cluster1 2023-08-26T13:53:32.428 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ eval instance=0 2023-08-26T13:53:32.428 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++++ instance=0 2023-08-26T13:53:32.428 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2023-08-26T13:53:32.442 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-08-26T13:53:32.442 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-08-26T13:53:32.442 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-08-26T13:53:32.442 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_14 2023-08-26T13:53:32.559 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:Invalid command: unused arguments: ['mirror/image_14'] 2023-08-26T13:53:32.560 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:rbd mirror status : get status for rbd mirror 2023-08-26T13:53:32.560 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:admin_socket: invalid command 2023-08-26T13:53:32.563 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-08-26T13:53:32.563 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_14 2023-08-26T13:53:32.563 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local cluster=cluster1:1 2023-08-26T13:53:32.563 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ shift 2023-08-26T13:53:32.563 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local instance 2023-08-26T13:53:32.564 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ set_cluster_instance cluster1:1 cluster instance 2023-08-26T13:53:32.564 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local val=cluster1:1 2023-08-26T13:53:32.564 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local cluster_var_name=cluster 2023-08-26T13:53:32.564 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local instance_var_name=instance 2023-08-26T13:53:32.564 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ cluster=cluster1 2023-08-26T13:53:32.565 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ instance=1 2023-08-26T13:53:32.565 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ '[' 1 = cluster1:1 ']' 2023-08-26T13:53:32.565 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ eval cluster=cluster1 2023-08-26T13:53:32.565 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ cluster=cluster1 2023-08-26T13:53:32.565 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ eval instance=1 2023-08-26T13:53:32.566 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ instance=1 2023-08-26T13:53:32.566 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ daemon_asok_file cluster1:1 cluster1 2023-08-26T13:53:32.566 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ local local_cluster=cluster1:1 2023-08-26T13:53:32.566 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ local cluster=cluster1 2023-08-26T13:53:32.566 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ local instance 2023-08-26T13:53:32.566 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2023-08-26T13:53:32.567 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ local val=cluster1:1 2023-08-26T13:53:32.567 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ local cluster_var_name=local_cluster 2023-08-26T13:53:32.567 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ local instance_var_name=instance 2023-08-26T13:53:32.567 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ cluster=cluster1 2023-08-26T13:53:32.567 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ instance=1 2023-08-26T13:53:32.568 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ '[' 1 = cluster1:1 ']' 2023-08-26T13:53:32.568 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ eval local_cluster=cluster1 2023-08-26T13:53:32.568 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++++ local_cluster=cluster1 2023-08-26T13:53:32.569 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ eval instance=1 2023-08-26T13:53:32.569 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++++ instance=1 2023-08-26T13:53:32.569 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2023-08-26T13:53:32.582 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-08-26T13:53:32.582 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-08-26T13:53:32.583 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-08-26T13:53:32.583 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_14 2023-08-26T13:53:32.696 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ return 0 2023-08-26T13:53:32.697 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ status_result=' "state": "Replaying"' 2023-08-26T13:53:32.697 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ echo ' "state": "Replaying"' 2023-08-26T13:53:32.697 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ grep -i Replaying 2023-08-26T13:53:32.698 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: "state": "Replaying" 2023-08-26T13:53:32.700 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ current_state=started 2023-08-26T13:53:32.700 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ test started = started 2023-08-26T13:53:32.700 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ return 0 2023-08-26T13:53:32.700 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_14 2023-08-26T13:53:32.701 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local local_cluster=cluster1 2023-08-26T13:53:32.701 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster2 2023-08-26T13:53:32.701 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T13:53:32.701 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=image_14 2023-08-26T13:53:32.701 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ '[' snapshot = journal ']' 2023-08-26T13:53:32.702 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ '[' snapshot = snapshot ']' 2023-08-26T13:53:32.702 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_14 2023-08-26T13:53:32.702 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local local_cluster=cluster1 2023-08-26T13:53:32.702 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster2 2023-08-26T13:53:32.703 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T13:53:32.703 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=image_14 2023-08-26T13:53:32.703 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ mkfname cluster2-mirror-image_14.status 2023-08-26T13:53:32.703 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ echo cluster2-mirror-image_14.status 2023-08-26T13:53:32.703 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ sed -e 's|[/ ]|_|g' 2023-08-26T13:53:32.704 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local status_log=/tmp/tmp.TfDn6BcQ3N/cluster2-mirror-image_14.status 2023-08-26T13:53:32.704 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ mkfname cluster1-mirror-image_14.status 2023-08-26T13:53:32.705 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ echo cluster1-mirror-image_14.status 2023-08-26T13:53:32.705 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ sed -e 's|[/ ]|_|g' 2023-08-26T13:53:32.705 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local local_status_log=/tmp/tmp.TfDn6BcQ3N/cluster1-mirror-image_14.status 2023-08-26T13:53:32.706 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ mirror_image_snapshot cluster2 mirror image_14 2023-08-26T13:53:32.706 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster2 2023-08-26T13:53:32.706 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T13:53:32.706 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=image_14 2023-08-26T13:53:32.707 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_14 2023-08-26T13:53:33.416 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:Snapshot ID: 145 2023-08-26T13:53:33.426 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_14 /tmp/tmp.TfDn6BcQ3N/cluster2-mirror-image_14.status 2023-08-26T13:53:33.427 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster2 2023-08-26T13:53:33.427 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T13:53:33.427 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=image_14 2023-08-26T13:53:33.427 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local log=/tmp/tmp.TfDn6BcQ3N/cluster2-mirror-image_14.status 2023-08-26T13:53:33.427 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_14 --format xml 2023-08-26T13:53:33.428 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-08-26T13:53:33.471 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ xmlstarlet sel -t -v //snapshot/id 2023-08-26T13:53:33.473 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local snapshot_id=145 2023-08-26T13:53:33.473 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ true 2023-08-26T13:53:33.473 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-08-26T13:53:33.473 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ sleep 0.2 2023-08-26T13:53:33.674 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_14 /tmp/tmp.TfDn6BcQ3N/cluster1-mirror-image_14.status 2023-08-26T13:53:33.675 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster1 2023-08-26T13:53:33.675 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T13:53:33.675 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=image_14 2023-08-26T13:53:33.675 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local log=/tmp/tmp.TfDn6BcQ3N/cluster1-mirror-image_14.status 2023-08-26T13:53:33.675 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_14 --format xml 2023-08-26T13:53:33.676 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-08-26T13:53:33.726 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ true 2023-08-26T13:53:33.726 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-08-26T13:53:33.727 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:-:1.1: Document is empty 2023-08-26T13:53:33.728 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr: 2023-08-26T13:53:33.728 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:^ 2023-08-26T13:53:33.729 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local primary_snapshot_id= 2023-08-26T13:53:33.729 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ test 145 = '' 2023-08-26T13:53:33.729 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-08-26T13:53:33.729 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ sleep 0.4 2023-08-26T13:53:34.130 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_14 /tmp/tmp.TfDn6BcQ3N/cluster1-mirror-image_14.status 2023-08-26T13:53:34.130 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster1 2023-08-26T13:53:34.130 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T13:53:34.130 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=image_14 2023-08-26T13:53:34.131 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local log=/tmp/tmp.TfDn6BcQ3N/cluster1-mirror-image_14.status 2023-08-26T13:53:34.131 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_14 --format xml 2023-08-26T13:53:34.131 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-08-26T13:53:34.177 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ true 2023-08-26T13:53:34.178 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-08-26T13:53:34.179 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:-:1.1: Document is empty 2023-08-26T13:53:34.179 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr: 2023-08-26T13:53:34.179 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:^ 2023-08-26T13:53:34.180 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local primary_snapshot_id= 2023-08-26T13:53:34.180 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ test 145 = '' 2023-08-26T13:53:34.180 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-08-26T13:53:34.180 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ sleep 0.8 2023-08-26T13:53:34.981 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_14 /tmp/tmp.TfDn6BcQ3N/cluster1-mirror-image_14.status 2023-08-26T13:53:34.981 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster1 2023-08-26T13:53:34.981 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T13:53:34.981 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=image_14 2023-08-26T13:53:34.982 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local log=/tmp/tmp.TfDn6BcQ3N/cluster1-mirror-image_14.status 2023-08-26T13:53:34.982 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_14 --format xml 2023-08-26T13:53:34.982 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-08-26T13:53:35.025 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-08-26T13:53:35.027 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local primary_snapshot_id=145 2023-08-26T13:53:35.027 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ test 145 = 145 2023-08-26T13:53:35.027 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ return 0 2023-08-26T13:53:35.028 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ wait_for_snap_present cluster1 mirror image_14 snap 2023-08-26T13:53:35.028 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster1 2023-08-26T13:53:35.028 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T13:53:35.028 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=image_14 2023-08-26T13:53:35.028 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local snap_name=snap 2023-08-26T13:53:35.029 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local s 2023-08-26T13:53:35.029 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2023-08-26T13:53:35.029 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ sleep 1 2023-08-26T13:53:36.028 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rbd --cluster cluster1 info mirror/image_14@snap 2023-08-26T13:53:36.075 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:rbd image 'image_14': 2023-08-26T13:53:36.075 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: size 128 MiB in 32 objects 2023-08-26T13:53:36.075 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: order 22 (4 MiB objects) 2023-08-26T13:53:36.076 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: snapshot_count: 2 2023-08-26T13:53:36.076 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: id: 11d2a5594f34 2023-08-26T13:53:36.076 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: block_name_prefix: rbd_data.11d2a5594f34 2023-08-26T13:53:36.076 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: format: 2 2023-08-26T13:53:36.076 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: features: layering, exclusive-lock, non-primary 2023-08-26T13:53:36.077 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: op_features: 2023-08-26T13:53:36.077 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: flags: 2023-08-26T13:53:36.077 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: create_timestamp: Sat Aug 26 13:44:59 2023 2023-08-26T13:53:36.077 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: access_timestamp: Sat Aug 26 13:44:59 2023 2023-08-26T13:53:36.077 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: modify_timestamp: Sat Aug 26 13:44:59 2023 2023-08-26T13:53:36.078 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: protected: False 2023-08-26T13:53:36.078 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: mirroring state: enabled 2023-08-26T13:53:36.078 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: mirroring mode: snapshot 2023-08-26T13:53:36.078 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: mirroring global id: 0fecd808-a8ef-45b7-aa0b-52385a8e2b1e 2023-08-26T13:53:36.078 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: mirroring primary: false 2023-08-26T13:53:36.080 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ return 0 2023-08-26T13:53:36.080 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ compare_image_snaps mirror image_14 snap 2023-08-26T13:53:36.080 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T13:53:36.080 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=image_14 2023-08-26T13:53:36.081 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local snap_name=snap 2023-08-26T13:53:36.081 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local ret=0 2023-08-26T13:53:36.081 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local rmt_export=/tmp/tmp.TfDn6BcQ3N/cluster2-mirror-image_14.export 2023-08-26T13:53:36.082 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local loc_export=/tmp/tmp.TfDn6BcQ3N/cluster1-mirror-image_14.export 2023-08-26T13:53:36.082 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rm -f /tmp/tmp.TfDn6BcQ3N/cluster2-mirror-image_14.export /tmp/tmp.TfDn6BcQ3N/cluster1-mirror-image_14.export 2023-08-26T13:53:36.082 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rbd --cluster cluster2 -p mirror export image_14@snap /tmp/tmp.TfDn6BcQ3N/cluster2-mirror-image_14.export 2023-08-26T13:53:36.262 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-08-26T13:53:36.269 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rbd --cluster cluster1 -p mirror export image_14@snap /tmp/tmp.TfDn6BcQ3N/cluster1-mirror-image_14.export 2023-08-26T13:53:36.521 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-08-26T13:53:36.526 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ cmp /tmp/tmp.TfDn6BcQ3N/cluster2-mirror-image_14.export /tmp/tmp.TfDn6BcQ3N/cluster1-mirror-image_14.export 2023-08-26T13:53:36.581 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rm -f /tmp/tmp.TfDn6BcQ3N/cluster2-mirror-image_14.export /tmp/tmp.TfDn6BcQ3N/cluster1-mirror-image_14.export 2023-08-26T13:53:36.622 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ return 0 2023-08-26T13:53:36.622 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-08-26T13:53:36.622 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ image=image_15 2023-08-26T13:53:36.622 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ create_snap cluster2 mirror image_15 snap 2023-08-26T13:53:36.623 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster2 2023-08-26T13:53:36.623 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T13:53:36.623 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=image_15 2023-08-26T13:53:36.623 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local snap_name=snap 2023-08-26T13:53:36.623 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rbd --cluster cluster2 -p mirror snap create image_15@snap --debug-rbd=20 --debug-journaler=20 2023-08-26T13:53:38.427 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ wait_for_image_replay_started cluster1 mirror image_15 2023-08-26T13:53:38.427 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster1 2023-08-26T13:53:38.428 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T13:53:38.428 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=image_15 2023-08-26T13:53:38.428 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ wait_for_image_replay_state cluster1 mirror image_15 started 2023-08-26T13:53:38.428 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster1 2023-08-26T13:53:38.430 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T13:53:38.430 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=image_15 2023-08-26T13:53:38.430 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local state=started 2023-08-26T13:53:38.431 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local s 2023-08-26T13:53:38.431 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-08-26T13:53:38.431 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ sleep 1 2023-08-26T13:53:39.428 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ test_image_replay_state cluster1 mirror image_15 started 2023-08-26T13:53:39.429 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster1 2023-08-26T13:53:39.429 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T13:53:39.429 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=image_15 2023-08-26T13:53:39.429 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local test_state=started 2023-08-26T13:53:39.429 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local status_result 2023-08-26T13:53:39.430 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local current_state=stopped 2023-08-26T13:53:39.432 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_15 2023-08-26T13:53:39.433 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local cluster_instance=cluster1 2023-08-26T13:53:39.433 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ shift 2023-08-26T13:53:39.433 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ grep -i state 2023-08-26T13:53:39.433 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local cluster=cluster1 2023-08-26T13:53:39.433 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local instance=cluster1 2023-08-26T13:53:39.434 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local loop_instance 2023-08-26T13:53:39.434 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-08-26T13:53:39.434 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ sleep 0 2023-08-26T13:53:39.434 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ '[' cluster1 '!=' cluster1 ']' 2023-08-26T13:53:39.434 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ seq 0 3 2023-08-26T13:53:39.434 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-08-26T13:53:39.435 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_15 2023-08-26T13:53:39.435 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local cluster=cluster1:0 2023-08-26T13:53:39.435 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ shift 2023-08-26T13:53:39.435 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local instance 2023-08-26T13:53:39.435 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ set_cluster_instance cluster1:0 cluster instance 2023-08-26T13:53:39.436 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local val=cluster1:0 2023-08-26T13:53:39.436 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local cluster_var_name=cluster 2023-08-26T13:53:39.436 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local instance_var_name=instance 2023-08-26T13:53:39.436 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ cluster=cluster1 2023-08-26T13:53:39.436 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ instance=0 2023-08-26T13:53:39.436 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ '[' 0 = cluster1:0 ']' 2023-08-26T13:53:39.437 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ eval cluster=cluster1 2023-08-26T13:53:39.437 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ cluster=cluster1 2023-08-26T13:53:39.437 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ eval instance=0 2023-08-26T13:53:39.437 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ instance=0 2023-08-26T13:53:39.437 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ daemon_asok_file cluster1:0 cluster1 2023-08-26T13:53:39.438 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ local local_cluster=cluster1:0 2023-08-26T13:53:39.438 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ local cluster=cluster1 2023-08-26T13:53:39.438 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ local instance 2023-08-26T13:53:39.439 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2023-08-26T13:53:39.439 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ local val=cluster1:0 2023-08-26T13:53:39.440 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ local cluster_var_name=local_cluster 2023-08-26T13:53:39.440 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ local instance_var_name=instance 2023-08-26T13:53:39.440 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ cluster=cluster1 2023-08-26T13:53:39.440 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ instance=0 2023-08-26T13:53:39.440 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ '[' 0 = cluster1:0 ']' 2023-08-26T13:53:39.441 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ eval local_cluster=cluster1 2023-08-26T13:53:39.441 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++++ local_cluster=cluster1 2023-08-26T13:53:39.441 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ eval instance=0 2023-08-26T13:53:39.441 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++++ instance=0 2023-08-26T13:53:39.441 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2023-08-26T13:53:39.455 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-08-26T13:53:39.455 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-08-26T13:53:39.455 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-08-26T13:53:39.456 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_15 2023-08-26T13:53:39.571 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:Invalid command: unused arguments: ['mirror/image_15'] 2023-08-26T13:53:39.571 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:rbd mirror status : get status for rbd mirror 2023-08-26T13:53:39.572 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:admin_socket: invalid command 2023-08-26T13:53:39.574 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-08-26T13:53:39.575 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_15 2023-08-26T13:53:39.575 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local cluster=cluster1:1 2023-08-26T13:53:39.575 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ shift 2023-08-26T13:53:39.575 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local instance 2023-08-26T13:53:39.576 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ set_cluster_instance cluster1:1 cluster instance 2023-08-26T13:53:39.576 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local val=cluster1:1 2023-08-26T13:53:39.576 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local cluster_var_name=cluster 2023-08-26T13:53:39.576 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local instance_var_name=instance 2023-08-26T13:53:39.576 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ cluster=cluster1 2023-08-26T13:53:39.577 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ instance=1 2023-08-26T13:53:39.577 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ '[' 1 = cluster1:1 ']' 2023-08-26T13:53:39.577 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ eval cluster=cluster1 2023-08-26T13:53:39.577 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ cluster=cluster1 2023-08-26T13:53:39.577 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ eval instance=1 2023-08-26T13:53:39.577 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ instance=1 2023-08-26T13:53:39.578 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ daemon_asok_file cluster1:1 cluster1 2023-08-26T13:53:39.578 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ local local_cluster=cluster1:1 2023-08-26T13:53:39.578 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ local cluster=cluster1 2023-08-26T13:53:39.578 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ local instance 2023-08-26T13:53:39.578 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2023-08-26T13:53:39.579 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ local val=cluster1:1 2023-08-26T13:53:39.579 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ local cluster_var_name=local_cluster 2023-08-26T13:53:39.579 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ local instance_var_name=instance 2023-08-26T13:53:39.579 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ cluster=cluster1 2023-08-26T13:53:39.579 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ instance=1 2023-08-26T13:53:39.580 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ '[' 1 = cluster1:1 ']' 2023-08-26T13:53:39.580 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ eval local_cluster=cluster1 2023-08-26T13:53:39.580 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++++ local_cluster=cluster1 2023-08-26T13:53:39.580 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ eval instance=1 2023-08-26T13:53:39.580 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++++ instance=1 2023-08-26T13:53:39.581 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2023-08-26T13:53:39.593 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-08-26T13:53:39.593 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-08-26T13:53:39.593 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-08-26T13:53:39.593 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_15 2023-08-26T13:53:39.704 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:Invalid command: unused arguments: ['mirror/image_15'] 2023-08-26T13:53:39.704 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:rbd mirror status : get status for rbd mirror 2023-08-26T13:53:39.704 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:admin_socket: invalid command 2023-08-26T13:53:39.708 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-08-26T13:53:39.708 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/image_15 2023-08-26T13:53:39.708 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local cluster=cluster1:2 2023-08-26T13:53:39.708 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ shift 2023-08-26T13:53:39.708 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local instance 2023-08-26T13:53:39.709 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ set_cluster_instance cluster1:2 cluster instance 2023-08-26T13:53:39.709 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local val=cluster1:2 2023-08-26T13:53:39.709 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local cluster_var_name=cluster 2023-08-26T13:53:39.709 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local instance_var_name=instance 2023-08-26T13:53:39.709 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ cluster=cluster1 2023-08-26T13:53:39.710 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ instance=2 2023-08-26T13:53:39.710 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ '[' 2 = cluster1:2 ']' 2023-08-26T13:53:39.710 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ eval cluster=cluster1 2023-08-26T13:53:39.710 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ cluster=cluster1 2023-08-26T13:53:39.711 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ eval instance=2 2023-08-26T13:53:39.711 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ instance=2 2023-08-26T13:53:39.711 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ daemon_asok_file cluster1:2 cluster1 2023-08-26T13:53:39.711 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ local local_cluster=cluster1:2 2023-08-26T13:53:39.711 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ local cluster=cluster1 2023-08-26T13:53:39.712 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ local instance 2023-08-26T13:53:39.712 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2023-08-26T13:53:39.712 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ local val=cluster1:2 2023-08-26T13:53:39.712 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ local cluster_var_name=local_cluster 2023-08-26T13:53:39.712 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ local instance_var_name=instance 2023-08-26T13:53:39.712 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ cluster=cluster1 2023-08-26T13:53:39.713 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ instance=2 2023-08-26T13:53:39.713 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ '[' 2 = cluster1:2 ']' 2023-08-26T13:53:39.713 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ eval local_cluster=cluster1 2023-08-26T13:53:39.713 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++++ local_cluster=cluster1 2023-08-26T13:53:39.713 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ eval instance=2 2023-08-26T13:53:39.713 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++++ instance=2 2023-08-26T13:53:39.714 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2023-08-26T13:53:39.725 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-08-26T13:53:39.726 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-08-26T13:53:39.726 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-08-26T13:53:39.726 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/image_15 2023-08-26T13:53:39.839 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ return 0 2023-08-26T13:53:39.840 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ status_result=' "state": "Replaying"' 2023-08-26T13:53:39.840 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ echo ' "state": "Replaying"' 2023-08-26T13:53:39.840 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ grep -i Replaying 2023-08-26T13:53:39.841 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: "state": "Replaying" 2023-08-26T13:53:39.843 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ current_state=started 2023-08-26T13:53:39.843 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ test started = started 2023-08-26T13:53:39.843 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ return 0 2023-08-26T13:53:39.843 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_15 2023-08-26T13:53:39.844 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local local_cluster=cluster1 2023-08-26T13:53:39.844 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster2 2023-08-26T13:53:39.844 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T13:53:39.844 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=image_15 2023-08-26T13:53:39.844 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ '[' snapshot = journal ']' 2023-08-26T13:53:39.845 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ '[' snapshot = snapshot ']' 2023-08-26T13:53:39.845 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_15 2023-08-26T13:53:39.845 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local local_cluster=cluster1 2023-08-26T13:53:39.845 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster2 2023-08-26T13:53:39.845 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T13:53:39.845 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=image_15 2023-08-26T13:53:39.846 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ mkfname cluster2-mirror-image_15.status 2023-08-26T13:53:39.846 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ echo cluster2-mirror-image_15.status 2023-08-26T13:53:39.846 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ sed -e 's|[/ ]|_|g' 2023-08-26T13:53:39.846 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local status_log=/tmp/tmp.TfDn6BcQ3N/cluster2-mirror-image_15.status 2023-08-26T13:53:39.846 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ mkfname cluster1-mirror-image_15.status 2023-08-26T13:53:39.847 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ echo cluster1-mirror-image_15.status 2023-08-26T13:53:39.847 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ sed -e 's|[/ ]|_|g' 2023-08-26T13:53:39.848 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local local_status_log=/tmp/tmp.TfDn6BcQ3N/cluster1-mirror-image_15.status 2023-08-26T13:53:39.848 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ mirror_image_snapshot cluster2 mirror image_15 2023-08-26T13:53:39.848 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster2 2023-08-26T13:53:39.848 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T13:53:39.849 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=image_15 2023-08-26T13:53:39.849 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_15 2023-08-26T13:53:40.431 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:Snapshot ID: 147 2023-08-26T13:53:40.440 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_15 /tmp/tmp.TfDn6BcQ3N/cluster2-mirror-image_15.status 2023-08-26T13:53:40.440 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster2 2023-08-26T13:53:40.440 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T13:53:40.441 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=image_15 2023-08-26T13:53:40.441 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local log=/tmp/tmp.TfDn6BcQ3N/cluster2-mirror-image_15.status 2023-08-26T13:53:40.441 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_15 --format xml 2023-08-26T13:53:40.442 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-08-26T13:53:40.483 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ xmlstarlet sel -t -v //snapshot/id 2023-08-26T13:53:40.485 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local snapshot_id=147 2023-08-26T13:53:40.485 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ true 2023-08-26T13:53:40.486 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-08-26T13:53:40.486 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ sleep 0.2 2023-08-26T13:53:40.686 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_15 /tmp/tmp.TfDn6BcQ3N/cluster1-mirror-image_15.status 2023-08-26T13:53:40.687 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster1 2023-08-26T13:53:40.687 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T13:53:40.687 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=image_15 2023-08-26T13:53:40.687 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local log=/tmp/tmp.TfDn6BcQ3N/cluster1-mirror-image_15.status 2023-08-26T13:53:40.688 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_15 --format xml 2023-08-26T13:53:40.688 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-08-26T13:53:40.734 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-08-26T13:53:40.736 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local primary_snapshot_id=82 2023-08-26T13:53:40.736 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ test 147 = 82 2023-08-26T13:53:40.736 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-08-26T13:53:40.736 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ sleep 0.4 2023-08-26T13:53:41.136 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_15 /tmp/tmp.TfDn6BcQ3N/cluster1-mirror-image_15.status 2023-08-26T13:53:41.137 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster1 2023-08-26T13:53:41.137 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T13:53:41.137 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=image_15 2023-08-26T13:53:41.137 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local log=/tmp/tmp.TfDn6BcQ3N/cluster1-mirror-image_15.status 2023-08-26T13:53:41.138 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_15 --format xml 2023-08-26T13:53:41.138 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-08-26T13:53:41.184 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-08-26T13:53:41.186 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local primary_snapshot_id=82 2023-08-26T13:53:41.186 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ test 147 = 82 2023-08-26T13:53:41.186 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-08-26T13:53:41.187 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ sleep 0.8 2023-08-26T13:53:41.987 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_15 /tmp/tmp.TfDn6BcQ3N/cluster1-mirror-image_15.status 2023-08-26T13:53:41.987 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster1 2023-08-26T13:53:41.988 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T13:53:41.988 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=image_15 2023-08-26T13:53:41.988 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local log=/tmp/tmp.TfDn6BcQ3N/cluster1-mirror-image_15.status 2023-08-26T13:53:41.988 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_15 --format xml 2023-08-26T13:53:41.988 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-08-26T13:53:42.038 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ true 2023-08-26T13:53:42.038 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-08-26T13:53:42.039 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:-:1.1: Document is empty 2023-08-26T13:53:42.039 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr: 2023-08-26T13:53:42.039 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:^ 2023-08-26T13:53:42.040 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local primary_snapshot_id= 2023-08-26T13:53:42.041 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ test 147 = '' 2023-08-26T13:53:42.041 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-08-26T13:53:42.041 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ sleep 1.6 2023-08-26T13:53:43.642 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_15 /tmp/tmp.TfDn6BcQ3N/cluster1-mirror-image_15.status 2023-08-26T13:53:43.642 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster1 2023-08-26T13:53:43.642 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T13:53:43.642 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=image_15 2023-08-26T13:53:43.642 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local log=/tmp/tmp.TfDn6BcQ3N/cluster1-mirror-image_15.status 2023-08-26T13:53:43.643 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_15 --format xml 2023-08-26T13:53:43.643 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-08-26T13:53:43.703 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-08-26T13:53:43.706 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local primary_snapshot_id=147 2023-08-26T13:53:43.706 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ test 147 = 147 2023-08-26T13:53:43.706 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ return 0 2023-08-26T13:53:43.706 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ wait_for_snap_present cluster1 mirror image_15 snap 2023-08-26T13:53:43.707 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster1 2023-08-26T13:53:43.707 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T13:53:43.707 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=image_15 2023-08-26T13:53:43.707 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local snap_name=snap 2023-08-26T13:53:43.707 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local s 2023-08-26T13:53:43.708 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2023-08-26T13:53:43.708 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ sleep 1 2023-08-26T13:53:44.706 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rbd --cluster cluster1 info mirror/image_15@snap 2023-08-26T13:53:44.753 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:rbd image 'image_15': 2023-08-26T13:53:44.754 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: size 128 MiB in 32 objects 2023-08-26T13:53:44.754 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: order 22 (4 MiB objects) 2023-08-26T13:53:44.754 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: snapshot_count: 2 2023-08-26T13:53:44.755 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: id: 122be9d75dc4 2023-08-26T13:53:44.755 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: block_name_prefix: rbd_data.122be9d75dc4 2023-08-26T13:53:44.755 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: format: 2 2023-08-26T13:53:44.755 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: features: layering, exclusive-lock, non-primary 2023-08-26T13:53:44.756 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: op_features: 2023-08-26T13:53:44.756 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: flags: 2023-08-26T13:53:44.756 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: create_timestamp: Sat Aug 26 13:45:11 2023 2023-08-26T13:53:44.757 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: access_timestamp: Sat Aug 26 13:45:11 2023 2023-08-26T13:53:44.757 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: modify_timestamp: Sat Aug 26 13:45:11 2023 2023-08-26T13:53:44.757 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: protected: False 2023-08-26T13:53:44.757 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: mirroring state: enabled 2023-08-26T13:53:44.758 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: mirroring mode: snapshot 2023-08-26T13:53:44.758 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: mirroring global id: 57b5e313-8920-4ded-8ea4-ab852f9e42c2 2023-08-26T13:53:44.759 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: mirroring primary: false 2023-08-26T13:53:44.760 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ return 0 2023-08-26T13:53:44.760 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ compare_image_snaps mirror image_15 snap 2023-08-26T13:53:44.760 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T13:53:44.761 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=image_15 2023-08-26T13:53:44.761 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local snap_name=snap 2023-08-26T13:53:44.761 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local ret=0 2023-08-26T13:53:44.761 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local rmt_export=/tmp/tmp.TfDn6BcQ3N/cluster2-mirror-image_15.export 2023-08-26T13:53:44.761 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local loc_export=/tmp/tmp.TfDn6BcQ3N/cluster1-mirror-image_15.export 2023-08-26T13:53:44.762 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rm -f /tmp/tmp.TfDn6BcQ3N/cluster2-mirror-image_15.export /tmp/tmp.TfDn6BcQ3N/cluster1-mirror-image_15.export 2023-08-26T13:53:44.762 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rbd --cluster cluster2 -p mirror export image_15@snap /tmp/tmp.TfDn6BcQ3N/cluster2-mirror-image_15.export 2023-08-26T13:53:44.933 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-08-26T13:53:44.939 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rbd --cluster cluster1 -p mirror export image_15@snap /tmp/tmp.TfDn6BcQ3N/cluster1-mirror-image_15.export 2023-08-26T13:53:45.186 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-08-26T13:53:45.192 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ cmp /tmp/tmp.TfDn6BcQ3N/cluster2-mirror-image_15.export /tmp/tmp.TfDn6BcQ3N/cluster1-mirror-image_15.export 2023-08-26T13:53:45.247 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rm -f /tmp/tmp.TfDn6BcQ3N/cluster2-mirror-image_15.export /tmp/tmp.TfDn6BcQ3N/cluster1-mirror-image_15.export 2023-08-26T13:53:45.284 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ return 0 2023-08-26T13:53:45.284 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-08-26T13:53:45.284 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ image=image_16 2023-08-26T13:53:45.284 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ create_snap cluster2 mirror image_16 snap 2023-08-26T13:53:45.284 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster2 2023-08-26T13:53:45.285 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T13:53:45.285 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=image_16 2023-08-26T13:53:45.285 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local snap_name=snap 2023-08-26T13:53:45.285 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rbd --cluster cluster2 -p mirror snap create image_16@snap --debug-rbd=20 --debug-journaler=20 2023-08-26T13:53:47.183 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ wait_for_image_replay_started cluster1 mirror image_16 2023-08-26T13:53:47.184 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster1 2023-08-26T13:53:47.184 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T13:53:47.184 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=image_16 2023-08-26T13:53:47.184 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ wait_for_image_replay_state cluster1 mirror image_16 started 2023-08-26T13:53:47.185 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster1 2023-08-26T13:53:47.185 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T13:53:47.185 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=image_16 2023-08-26T13:53:47.185 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local state=started 2023-08-26T13:53:47.185 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local s 2023-08-26T13:53:47.186 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-08-26T13:53:47.186 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ sleep 1 2023-08-26T13:53:48.185 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ test_image_replay_state cluster1 mirror image_16 started 2023-08-26T13:53:48.186 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster1 2023-08-26T13:53:48.186 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T13:53:48.186 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=image_16 2023-08-26T13:53:48.186 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local test_state=started 2023-08-26T13:53:48.186 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local status_result 2023-08-26T13:53:48.187 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local current_state=stopped 2023-08-26T13:53:48.187 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_16 2023-08-26T13:53:48.187 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local cluster_instance=cluster1 2023-08-26T13:53:48.187 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ shift 2023-08-26T13:53:48.188 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local cluster=cluster1 2023-08-26T13:53:48.188 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local instance=cluster1 2023-08-26T13:53:48.188 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local loop_instance 2023-08-26T13:53:48.188 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-08-26T13:53:48.188 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ grep -i state 2023-08-26T13:53:48.189 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ sleep 0 2023-08-26T13:53:48.192 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ '[' cluster1 '!=' cluster1 ']' 2023-08-26T13:53:48.192 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ seq 0 3 2023-08-26T13:53:48.192 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-08-26T13:53:48.192 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_16 2023-08-26T13:53:48.193 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local cluster=cluster1:0 2023-08-26T13:53:48.193 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ shift 2023-08-26T13:53:48.193 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local instance 2023-08-26T13:53:48.193 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ set_cluster_instance cluster1:0 cluster instance 2023-08-26T13:53:48.193 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local val=cluster1:0 2023-08-26T13:53:48.194 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local cluster_var_name=cluster 2023-08-26T13:53:48.194 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local instance_var_name=instance 2023-08-26T13:53:48.194 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ cluster=cluster1 2023-08-26T13:53:48.194 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ instance=0 2023-08-26T13:53:48.195 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ '[' 0 = cluster1:0 ']' 2023-08-26T13:53:48.195 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ eval cluster=cluster1 2023-08-26T13:53:48.195 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ cluster=cluster1 2023-08-26T13:53:48.195 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ eval instance=0 2023-08-26T13:53:48.195 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ instance=0 2023-08-26T13:53:48.196 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ daemon_asok_file cluster1:0 cluster1 2023-08-26T13:53:48.196 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ local local_cluster=cluster1:0 2023-08-26T13:53:48.196 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ local cluster=cluster1 2023-08-26T13:53:48.196 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ local instance 2023-08-26T13:53:48.196 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2023-08-26T13:53:48.197 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ local val=cluster1:0 2023-08-26T13:53:48.197 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ local cluster_var_name=local_cluster 2023-08-26T13:53:48.197 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ local instance_var_name=instance 2023-08-26T13:53:48.197 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ cluster=cluster1 2023-08-26T13:53:48.197 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ instance=0 2023-08-26T13:53:48.198 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ '[' 0 = cluster1:0 ']' 2023-08-26T13:53:48.200 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ eval local_cluster=cluster1 2023-08-26T13:53:48.200 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++++ local_cluster=cluster1 2023-08-26T13:53:48.201 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ eval instance=0 2023-08-26T13:53:48.201 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++++ instance=0 2023-08-26T13:53:48.201 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2023-08-26T13:53:48.211 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-08-26T13:53:48.212 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-08-26T13:53:48.212 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-08-26T13:53:48.212 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_16 2023-08-26T13:53:48.329 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ return 0 2023-08-26T13:53:48.330 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ status_result=' "state": "Replaying"' 2023-08-26T13:53:48.330 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ echo ' "state": "Replaying"' 2023-08-26T13:53:48.330 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ grep -i Replaying 2023-08-26T13:53:48.331 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: "state": "Replaying" 2023-08-26T13:53:48.332 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ current_state=started 2023-08-26T13:53:48.333 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ test started = started 2023-08-26T13:53:48.333 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ return 0 2023-08-26T13:53:48.333 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_16 2023-08-26T13:53:48.333 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local local_cluster=cluster1 2023-08-26T13:53:48.333 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster2 2023-08-26T13:53:48.334 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T13:53:48.334 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=image_16 2023-08-26T13:53:48.334 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ '[' snapshot = journal ']' 2023-08-26T13:53:48.334 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ '[' snapshot = snapshot ']' 2023-08-26T13:53:48.334 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_16 2023-08-26T13:53:48.335 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local local_cluster=cluster1 2023-08-26T13:53:48.335 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster2 2023-08-26T13:53:48.335 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T13:53:48.335 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=image_16 2023-08-26T13:53:48.335 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ mkfname cluster2-mirror-image_16.status 2023-08-26T13:53:48.336 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ echo cluster2-mirror-image_16.status 2023-08-26T13:53:48.336 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ sed -e 's|[/ ]|_|g' 2023-08-26T13:53:48.336 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local status_log=/tmp/tmp.TfDn6BcQ3N/cluster2-mirror-image_16.status 2023-08-26T13:53:48.336 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ mkfname cluster1-mirror-image_16.status 2023-08-26T13:53:48.337 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ echo cluster1-mirror-image_16.status 2023-08-26T13:53:48.337 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ sed -e 's|[/ ]|_|g' 2023-08-26T13:53:48.337 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local local_status_log=/tmp/tmp.TfDn6BcQ3N/cluster1-mirror-image_16.status 2023-08-26T13:53:48.338 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ mirror_image_snapshot cluster2 mirror image_16 2023-08-26T13:53:48.338 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster2 2023-08-26T13:53:48.338 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T13:53:48.338 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=image_16 2023-08-26T13:53:48.338 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_16 2023-08-26T13:53:49.190 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:Snapshot ID: 149 2023-08-26T13:53:49.202 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_16 /tmp/tmp.TfDn6BcQ3N/cluster2-mirror-image_16.status 2023-08-26T13:53:49.202 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster2 2023-08-26T13:53:49.203 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T13:53:49.203 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=image_16 2023-08-26T13:53:49.203 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local log=/tmp/tmp.TfDn6BcQ3N/cluster2-mirror-image_16.status 2023-08-26T13:53:49.203 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_16 --format xml 2023-08-26T13:53:49.203 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-08-26T13:53:49.247 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ xmlstarlet sel -t -v //snapshot/id 2023-08-26T13:53:49.249 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local snapshot_id=149 2023-08-26T13:53:49.249 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ true 2023-08-26T13:53:49.250 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-08-26T13:53:49.250 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ sleep 0.2 2023-08-26T13:53:49.451 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_16 /tmp/tmp.TfDn6BcQ3N/cluster1-mirror-image_16.status 2023-08-26T13:53:49.451 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster1 2023-08-26T13:53:49.452 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T13:53:49.452 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=image_16 2023-08-26T13:53:49.452 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local log=/tmp/tmp.TfDn6BcQ3N/cluster1-mirror-image_16.status 2023-08-26T13:53:49.452 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_16 --format xml 2023-08-26T13:53:49.452 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-08-26T13:53:49.505 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-08-26T13:53:49.507 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local primary_snapshot_id=83 2023-08-26T13:53:49.507 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ test 149 = 83 2023-08-26T13:53:49.508 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-08-26T13:53:49.508 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ sleep 0.4 2023-08-26T13:53:49.909 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_16 /tmp/tmp.TfDn6BcQ3N/cluster1-mirror-image_16.status 2023-08-26T13:53:49.909 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster1 2023-08-26T13:53:49.909 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T13:53:49.910 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=image_16 2023-08-26T13:53:49.910 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local log=/tmp/tmp.TfDn6BcQ3N/cluster1-mirror-image_16.status 2023-08-26T13:53:49.910 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_16 --format xml 2023-08-26T13:53:49.910 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-08-26T13:53:49.959 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ true 2023-08-26T13:53:49.960 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-08-26T13:53:49.961 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:-:1.1: Document is empty 2023-08-26T13:53:49.961 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr: 2023-08-26T13:53:49.961 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:^ 2023-08-26T13:53:49.964 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local primary_snapshot_id= 2023-08-26T13:53:49.964 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ test 149 = '' 2023-08-26T13:53:49.964 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-08-26T13:53:49.965 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ sleep 0.8 2023-08-26T13:53:50.763 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_16 /tmp/tmp.TfDn6BcQ3N/cluster1-mirror-image_16.status 2023-08-26T13:53:50.763 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster1 2023-08-26T13:53:50.764 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T13:53:50.764 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=image_16 2023-08-26T13:53:50.764 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local log=/tmp/tmp.TfDn6BcQ3N/cluster1-mirror-image_16.status 2023-08-26T13:53:50.764 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_16 --format xml 2023-08-26T13:53:50.764 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-08-26T13:53:50.808 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-08-26T13:53:50.810 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local primary_snapshot_id=149 2023-08-26T13:53:50.810 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ test 149 = 149 2023-08-26T13:53:50.811 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ return 0 2023-08-26T13:53:50.811 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ wait_for_snap_present cluster1 mirror image_16 snap 2023-08-26T13:53:50.811 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster1 2023-08-26T13:53:50.811 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T13:53:50.811 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=image_16 2023-08-26T13:53:50.812 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local snap_name=snap 2023-08-26T13:53:50.812 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local s 2023-08-26T13:53:50.812 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2023-08-26T13:53:50.812 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ sleep 1 2023-08-26T13:53:51.811 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rbd --cluster cluster1 info mirror/image_16@snap 2023-08-26T13:53:51.865 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:rbd image 'image_16': 2023-08-26T13:53:51.865 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: size 128 MiB in 32 objects 2023-08-26T13:53:51.865 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: order 22 (4 MiB objects) 2023-08-26T13:53:51.865 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: snapshot_count: 2 2023-08-26T13:53:51.866 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: id: 12b68f4c8f70 2023-08-26T13:53:51.866 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: block_name_prefix: rbd_data.12b68f4c8f70 2023-08-26T13:53:51.866 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: format: 2 2023-08-26T13:53:51.866 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: features: layering, exclusive-lock, non-primary 2023-08-26T13:53:51.866 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: op_features: 2023-08-26T13:53:51.867 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: flags: 2023-08-26T13:53:51.867 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: create_timestamp: Sat Aug 26 13:45:17 2023 2023-08-26T13:53:51.867 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: access_timestamp: Sat Aug 26 13:45:17 2023 2023-08-26T13:53:51.867 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: modify_timestamp: Sat Aug 26 13:45:17 2023 2023-08-26T13:53:51.867 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: protected: False 2023-08-26T13:53:51.868 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: mirroring state: enabled 2023-08-26T13:53:51.868 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: mirroring mode: snapshot 2023-08-26T13:53:51.868 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: mirroring global id: bd80dfca-7cec-4de6-be52-c58244873bd3 2023-08-26T13:53:51.868 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: mirroring primary: false 2023-08-26T13:53:51.870 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ return 0 2023-08-26T13:53:51.870 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ compare_image_snaps mirror image_16 snap 2023-08-26T13:53:51.870 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T13:53:51.870 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=image_16 2023-08-26T13:53:51.871 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local snap_name=snap 2023-08-26T13:53:51.871 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local ret=0 2023-08-26T13:53:51.871 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local rmt_export=/tmp/tmp.TfDn6BcQ3N/cluster2-mirror-image_16.export 2023-08-26T13:53:51.871 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local loc_export=/tmp/tmp.TfDn6BcQ3N/cluster1-mirror-image_16.export 2023-08-26T13:53:51.872 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rm -f /tmp/tmp.TfDn6BcQ3N/cluster2-mirror-image_16.export /tmp/tmp.TfDn6BcQ3N/cluster1-mirror-image_16.export 2023-08-26T13:53:51.872 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rbd --cluster cluster2 -p mirror export image_16@snap /tmp/tmp.TfDn6BcQ3N/cluster2-mirror-image_16.export 2023-08-26T13:53:52.028 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-08-26T13:53:52.033 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rbd --cluster cluster1 -p mirror export image_16@snap /tmp/tmp.TfDn6BcQ3N/cluster1-mirror-image_16.export 2023-08-26T13:53:52.264 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-08-26T13:53:52.269 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ cmp /tmp/tmp.TfDn6BcQ3N/cluster2-mirror-image_16.export /tmp/tmp.TfDn6BcQ3N/cluster1-mirror-image_16.export 2023-08-26T13:53:52.323 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rm -f /tmp/tmp.TfDn6BcQ3N/cluster2-mirror-image_16.export /tmp/tmp.TfDn6BcQ3N/cluster1-mirror-image_16.export 2023-08-26T13:53:52.360 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ return 0 2023-08-26T13:53:52.361 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-08-26T13:53:52.361 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ image=image_17 2023-08-26T13:53:52.361 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ create_snap cluster2 mirror image_17 snap 2023-08-26T13:53:52.361 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster2 2023-08-26T13:53:52.361 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T13:53:52.362 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=image_17 2023-08-26T13:53:52.362 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local snap_name=snap 2023-08-26T13:53:52.362 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rbd --cluster cluster2 -p mirror snap create image_17@snap --debug-rbd=20 --debug-journaler=20 2023-08-26T13:53:54.198 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ wait_for_image_replay_started cluster1 mirror image_17 2023-08-26T13:53:54.198 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster1 2023-08-26T13:53:54.199 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T13:53:54.199 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=image_17 2023-08-26T13:53:54.199 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ wait_for_image_replay_state cluster1 mirror image_17 started 2023-08-26T13:53:54.199 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster1 2023-08-26T13:53:54.200 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T13:53:54.200 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=image_17 2023-08-26T13:53:54.200 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local state=started 2023-08-26T13:53:54.200 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local s 2023-08-26T13:53:54.200 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-08-26T13:53:54.201 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ sleep 1 2023-08-26T13:53:55.201 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ test_image_replay_state cluster1 mirror image_17 started 2023-08-26T13:53:55.202 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster1 2023-08-26T13:53:55.202 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T13:53:55.202 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=image_17 2023-08-26T13:53:55.203 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local test_state=started 2023-08-26T13:53:55.203 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local status_result 2023-08-26T13:53:55.203 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local current_state=stopped 2023-08-26T13:53:55.204 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_17 2023-08-26T13:53:55.204 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local cluster_instance=cluster1 2023-08-26T13:53:55.204 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ shift 2023-08-26T13:53:55.205 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ grep -i state 2023-08-26T13:53:55.205 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local cluster=cluster1 2023-08-26T13:53:55.205 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local instance=cluster1 2023-08-26T13:53:55.206 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local loop_instance 2023-08-26T13:53:55.206 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-08-26T13:53:55.206 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ sleep 0 2023-08-26T13:53:55.207 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ '[' cluster1 '!=' cluster1 ']' 2023-08-26T13:53:55.207 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ seq 0 3 2023-08-26T13:53:55.212 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-08-26T13:53:55.212 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_17 2023-08-26T13:53:55.213 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local cluster=cluster1:0 2023-08-26T13:53:55.213 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ shift 2023-08-26T13:53:55.213 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local instance 2023-08-26T13:53:55.214 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ set_cluster_instance cluster1:0 cluster instance 2023-08-26T13:53:55.214 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local val=cluster1:0 2023-08-26T13:53:55.214 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local cluster_var_name=cluster 2023-08-26T13:53:55.215 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local instance_var_name=instance 2023-08-26T13:53:55.215 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ cluster=cluster1 2023-08-26T13:53:55.215 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ instance=0 2023-08-26T13:53:55.216 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ '[' 0 = cluster1:0 ']' 2023-08-26T13:53:55.216 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ eval cluster=cluster1 2023-08-26T13:53:55.216 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ cluster=cluster1 2023-08-26T13:53:55.217 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ eval instance=0 2023-08-26T13:53:55.217 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ instance=0 2023-08-26T13:53:55.217 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ daemon_asok_file cluster1:0 cluster1 2023-08-26T13:53:55.218 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ local local_cluster=cluster1:0 2023-08-26T13:53:55.218 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ local cluster=cluster1 2023-08-26T13:53:55.218 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ local instance 2023-08-26T13:53:55.219 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2023-08-26T13:53:55.219 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ local val=cluster1:0 2023-08-26T13:53:55.219 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ local cluster_var_name=local_cluster 2023-08-26T13:53:55.220 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ local instance_var_name=instance 2023-08-26T13:53:55.220 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ cluster=cluster1 2023-08-26T13:53:55.220 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ instance=0 2023-08-26T13:53:55.221 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ '[' 0 = cluster1:0 ']' 2023-08-26T13:53:55.221 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ eval local_cluster=cluster1 2023-08-26T13:53:55.221 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++++ local_cluster=cluster1 2023-08-26T13:53:55.222 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ eval instance=0 2023-08-26T13:53:55.222 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++++ instance=0 2023-08-26T13:53:55.222 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2023-08-26T13:53:55.226 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-08-26T13:53:55.226 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-08-26T13:53:55.226 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-08-26T13:53:55.227 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_17 2023-08-26T13:53:55.337 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:Invalid command: unused arguments: ['mirror/image_17'] 2023-08-26T13:53:55.337 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:rbd mirror status : get status for rbd mirror 2023-08-26T13:53:55.337 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:admin_socket: invalid command 2023-08-26T13:53:55.341 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-08-26T13:53:55.341 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_17 2023-08-26T13:53:55.341 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local cluster=cluster1:1 2023-08-26T13:53:55.341 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ shift 2023-08-26T13:53:55.341 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local instance 2023-08-26T13:53:55.342 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ set_cluster_instance cluster1:1 cluster instance 2023-08-26T13:53:55.342 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local val=cluster1:1 2023-08-26T13:53:55.342 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local cluster_var_name=cluster 2023-08-26T13:53:55.342 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local instance_var_name=instance 2023-08-26T13:53:55.342 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ cluster=cluster1 2023-08-26T13:53:55.343 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ instance=1 2023-08-26T13:53:55.343 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ '[' 1 = cluster1:1 ']' 2023-08-26T13:53:55.343 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ eval cluster=cluster1 2023-08-26T13:53:55.343 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ cluster=cluster1 2023-08-26T13:53:55.343 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ eval instance=1 2023-08-26T13:53:55.344 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ instance=1 2023-08-26T13:53:55.344 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ daemon_asok_file cluster1:1 cluster1 2023-08-26T13:53:55.344 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ local local_cluster=cluster1:1 2023-08-26T13:53:55.344 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ local cluster=cluster1 2023-08-26T13:53:55.345 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ local instance 2023-08-26T13:53:55.345 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2023-08-26T13:53:55.345 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ local val=cluster1:1 2023-08-26T13:53:55.345 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ local cluster_var_name=local_cluster 2023-08-26T13:53:55.345 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ local instance_var_name=instance 2023-08-26T13:53:55.346 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ cluster=cluster1 2023-08-26T13:53:55.346 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ instance=1 2023-08-26T13:53:55.346 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ '[' 1 = cluster1:1 ']' 2023-08-26T13:53:55.346 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ eval local_cluster=cluster1 2023-08-26T13:53:55.346 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++++ local_cluster=cluster1 2023-08-26T13:53:55.346 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ eval instance=1 2023-08-26T13:53:55.347 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++++ instance=1 2023-08-26T13:53:55.347 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2023-08-26T13:53:55.359 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-08-26T13:53:55.359 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-08-26T13:53:55.359 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-08-26T13:53:55.359 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_17 2023-08-26T13:53:55.465 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:Invalid command: unused arguments: ['mirror/image_17'] 2023-08-26T13:53:55.465 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:rbd mirror status : get status for rbd mirror 2023-08-26T13:53:55.465 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:admin_socket: invalid command 2023-08-26T13:53:55.468 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-08-26T13:53:55.469 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/image_17 2023-08-26T13:53:55.469 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local cluster=cluster1:2 2023-08-26T13:53:55.469 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ shift 2023-08-26T13:53:55.469 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local instance 2023-08-26T13:53:55.469 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ set_cluster_instance cluster1:2 cluster instance 2023-08-26T13:53:55.469 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local val=cluster1:2 2023-08-26T13:53:55.470 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local cluster_var_name=cluster 2023-08-26T13:53:55.470 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local instance_var_name=instance 2023-08-26T13:53:55.470 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ cluster=cluster1 2023-08-26T13:53:55.470 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ instance=2 2023-08-26T13:53:55.470 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ '[' 2 = cluster1:2 ']' 2023-08-26T13:53:55.471 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ eval cluster=cluster1 2023-08-26T13:53:55.471 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ cluster=cluster1 2023-08-26T13:53:55.471 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ eval instance=2 2023-08-26T13:53:55.471 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ instance=2 2023-08-26T13:53:55.471 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ daemon_asok_file cluster1:2 cluster1 2023-08-26T13:53:55.471 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ local local_cluster=cluster1:2 2023-08-26T13:53:55.472 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ local cluster=cluster1 2023-08-26T13:53:55.472 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ local instance 2023-08-26T13:53:55.472 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2023-08-26T13:53:55.472 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ local val=cluster1:2 2023-08-26T13:53:55.472 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ local cluster_var_name=local_cluster 2023-08-26T13:53:55.473 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ local instance_var_name=instance 2023-08-26T13:53:55.473 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ cluster=cluster1 2023-08-26T13:53:55.473 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ instance=2 2023-08-26T13:53:55.473 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ '[' 2 = cluster1:2 ']' 2023-08-26T13:53:55.473 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ eval local_cluster=cluster1 2023-08-26T13:53:55.473 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++++ local_cluster=cluster1 2023-08-26T13:53:55.474 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ eval instance=2 2023-08-26T13:53:55.474 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++++ instance=2 2023-08-26T13:53:55.474 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2023-08-26T13:53:55.486 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-08-26T13:53:55.487 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-08-26T13:53:55.487 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-08-26T13:53:55.487 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/image_17 2023-08-26T13:53:55.600 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ return 0 2023-08-26T13:53:55.601 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ status_result=' "state": "Replaying"' 2023-08-26T13:53:55.601 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ echo ' "state": "Replaying"' 2023-08-26T13:53:55.602 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ grep -i Replaying 2023-08-26T13:53:55.602 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: "state": "Replaying" 2023-08-26T13:53:55.604 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ current_state=started 2023-08-26T13:53:55.604 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ test started = started 2023-08-26T13:53:55.604 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ return 0 2023-08-26T13:53:55.605 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_17 2023-08-26T13:53:55.605 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local local_cluster=cluster1 2023-08-26T13:53:55.605 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster2 2023-08-26T13:53:55.605 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T13:53:55.605 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=image_17 2023-08-26T13:53:55.606 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ '[' snapshot = journal ']' 2023-08-26T13:53:55.606 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ '[' snapshot = snapshot ']' 2023-08-26T13:53:55.606 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_17 2023-08-26T13:53:55.606 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local local_cluster=cluster1 2023-08-26T13:53:55.606 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster2 2023-08-26T13:53:55.607 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T13:53:55.607 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=image_17 2023-08-26T13:53:55.607 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ mkfname cluster2-mirror-image_17.status 2023-08-26T13:53:55.607 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ echo cluster2-mirror-image_17.status 2023-08-26T13:53:55.607 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ sed -e 's|[/ ]|_|g' 2023-08-26T13:53:55.608 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local status_log=/tmp/tmp.TfDn6BcQ3N/cluster2-mirror-image_17.status 2023-08-26T13:53:55.608 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ mkfname cluster1-mirror-image_17.status 2023-08-26T13:53:55.608 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ echo cluster1-mirror-image_17.status 2023-08-26T13:53:55.608 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ sed -e 's|[/ ]|_|g' 2023-08-26T13:53:55.609 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local local_status_log=/tmp/tmp.TfDn6BcQ3N/cluster1-mirror-image_17.status 2023-08-26T13:53:55.609 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ mirror_image_snapshot cluster2 mirror image_17 2023-08-26T13:53:55.609 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster2 2023-08-26T13:53:55.610 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T13:53:55.611 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=image_17 2023-08-26T13:53:55.611 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_17 2023-08-26T13:53:56.201 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:Snapshot ID: 151 2023-08-26T13:53:56.210 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_17 /tmp/tmp.TfDn6BcQ3N/cluster2-mirror-image_17.status 2023-08-26T13:53:56.210 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster2 2023-08-26T13:53:56.211 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T13:53:56.211 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=image_17 2023-08-26T13:53:56.211 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local log=/tmp/tmp.TfDn6BcQ3N/cluster2-mirror-image_17.status 2023-08-26T13:53:56.211 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_17 --format xml 2023-08-26T13:53:56.212 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-08-26T13:53:56.253 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ xmlstarlet sel -t -v //snapshot/id 2023-08-26T13:53:56.255 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local snapshot_id=151 2023-08-26T13:53:56.255 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ true 2023-08-26T13:53:56.255 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-08-26T13:53:56.256 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ sleep 0.2 2023-08-26T13:53:56.456 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_17 /tmp/tmp.TfDn6BcQ3N/cluster1-mirror-image_17.status 2023-08-26T13:53:56.457 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster1 2023-08-26T13:53:56.457 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T13:53:56.457 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=image_17 2023-08-26T13:53:56.458 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local log=/tmp/tmp.TfDn6BcQ3N/cluster1-mirror-image_17.status 2023-08-26T13:53:56.458 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_17 --format xml 2023-08-26T13:53:56.458 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-08-26T13:53:56.506 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-08-26T13:53:56.508 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local primary_snapshot_id=84 2023-08-26T13:53:56.509 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ test 151 = 84 2023-08-26T13:53:56.509 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-08-26T13:53:56.509 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ sleep 0.4 2023-08-26T13:53:56.909 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_17 /tmp/tmp.TfDn6BcQ3N/cluster1-mirror-image_17.status 2023-08-26T13:53:56.910 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster1 2023-08-26T13:53:56.910 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T13:53:56.910 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=image_17 2023-08-26T13:53:56.910 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local log=/tmp/tmp.TfDn6BcQ3N/cluster1-mirror-image_17.status 2023-08-26T13:53:56.911 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_17 --format xml 2023-08-26T13:53:56.911 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-08-26T13:53:56.956 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ true 2023-08-26T13:53:56.957 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-08-26T13:53:56.958 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:-:1.1: Document is empty 2023-08-26T13:53:56.958 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr: 2023-08-26T13:53:56.958 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:^ 2023-08-26T13:53:56.959 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local primary_snapshot_id= 2023-08-26T13:53:56.959 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ test 151 = '' 2023-08-26T13:53:56.959 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-08-26T13:53:56.959 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ sleep 0.8 2023-08-26T13:53:57.760 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_17 /tmp/tmp.TfDn6BcQ3N/cluster1-mirror-image_17.status 2023-08-26T13:53:57.760 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster1 2023-08-26T13:53:57.760 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T13:53:57.761 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=image_17 2023-08-26T13:53:57.761 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local log=/tmp/tmp.TfDn6BcQ3N/cluster1-mirror-image_17.status 2023-08-26T13:53:57.761 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_17 --format xml 2023-08-26T13:53:57.762 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-08-26T13:53:57.833 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-08-26T13:53:57.836 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local primary_snapshot_id=151 2023-08-26T13:53:57.836 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ test 151 = 151 2023-08-26T13:53:57.837 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ return 0 2023-08-26T13:53:57.837 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ wait_for_snap_present cluster1 mirror image_17 snap 2023-08-26T13:53:57.837 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster1 2023-08-26T13:53:57.837 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T13:53:57.837 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=image_17 2023-08-26T13:53:57.838 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local snap_name=snap 2023-08-26T13:53:57.838 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local s 2023-08-26T13:53:57.838 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2023-08-26T13:53:57.838 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ sleep 1 2023-08-26T13:53:58.836 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rbd --cluster cluster1 info mirror/image_17@snap 2023-08-26T13:53:58.880 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:rbd image 'image_17': 2023-08-26T13:53:58.880 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: size 128 MiB in 32 objects 2023-08-26T13:53:58.881 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: order 22 (4 MiB objects) 2023-08-26T13:53:58.881 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: snapshot_count: 2 2023-08-26T13:53:58.881 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: id: 10b7e980d2ef 2023-08-26T13:53:58.881 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: block_name_prefix: rbd_data.10b7e980d2ef 2023-08-26T13:53:58.881 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: format: 2 2023-08-26T13:53:58.882 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: features: layering, exclusive-lock, non-primary 2023-08-26T13:53:58.882 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: op_features: 2023-08-26T13:53:58.882 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: flags: 2023-08-26T13:53:58.882 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: create_timestamp: Sat Aug 26 13:45:26 2023 2023-08-26T13:53:58.882 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: access_timestamp: Sat Aug 26 13:45:26 2023 2023-08-26T13:53:58.883 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: modify_timestamp: Sat Aug 26 13:45:26 2023 2023-08-26T13:53:58.883 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: protected: False 2023-08-26T13:53:58.883 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: mirroring state: enabled 2023-08-26T13:53:58.883 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: mirroring mode: snapshot 2023-08-26T13:53:58.883 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: mirroring global id: ec21e764-3783-49f1-b802-01400de8810c 2023-08-26T13:53:58.884 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: mirroring primary: false 2023-08-26T13:53:58.885 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ return 0 2023-08-26T13:53:58.885 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ compare_image_snaps mirror image_17 snap 2023-08-26T13:53:58.885 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T13:53:58.886 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=image_17 2023-08-26T13:53:58.886 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local snap_name=snap 2023-08-26T13:53:58.886 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local ret=0 2023-08-26T13:53:58.886 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local rmt_export=/tmp/tmp.TfDn6BcQ3N/cluster2-mirror-image_17.export 2023-08-26T13:53:58.887 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local loc_export=/tmp/tmp.TfDn6BcQ3N/cluster1-mirror-image_17.export 2023-08-26T13:53:58.887 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rm -f /tmp/tmp.TfDn6BcQ3N/cluster2-mirror-image_17.export /tmp/tmp.TfDn6BcQ3N/cluster1-mirror-image_17.export 2023-08-26T13:53:58.887 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rbd --cluster cluster2 -p mirror export image_17@snap /tmp/tmp.TfDn6BcQ3N/cluster2-mirror-image_17.export 2023-08-26T13:53:59.059 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-08-26T13:53:59.066 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rbd --cluster cluster1 -p mirror export image_17@snap /tmp/tmp.TfDn6BcQ3N/cluster1-mirror-image_17.export 2023-08-26T13:53:59.301 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-08-26T13:53:59.308 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ cmp /tmp/tmp.TfDn6BcQ3N/cluster2-mirror-image_17.export /tmp/tmp.TfDn6BcQ3N/cluster1-mirror-image_17.export 2023-08-26T13:53:59.363 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rm -f /tmp/tmp.TfDn6BcQ3N/cluster2-mirror-image_17.export /tmp/tmp.TfDn6BcQ3N/cluster1-mirror-image_17.export 2023-08-26T13:53:59.401 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ return 0 2023-08-26T13:53:59.401 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-08-26T13:53:59.401 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ image=image_18 2023-08-26T13:53:59.402 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ create_snap cluster2 mirror image_18 snap 2023-08-26T13:53:59.402 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster2 2023-08-26T13:53:59.402 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T13:53:59.402 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=image_18 2023-08-26T13:53:59.403 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local snap_name=snap 2023-08-26T13:53:59.403 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rbd --cluster cluster2 -p mirror snap create image_18@snap --debug-rbd=20 --debug-journaler=20 2023-08-26T13:54:01.218 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ wait_for_image_replay_started cluster1 mirror image_18 2023-08-26T13:54:01.218 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster1 2023-08-26T13:54:01.218 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T13:54:01.219 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=image_18 2023-08-26T13:54:01.219 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ wait_for_image_replay_state cluster1 mirror image_18 started 2023-08-26T13:54:01.219 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster1 2023-08-26T13:54:01.219 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T13:54:01.219 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=image_18 2023-08-26T13:54:01.220 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local state=started 2023-08-26T13:54:01.220 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local s 2023-08-26T13:54:01.220 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-08-26T13:54:01.220 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ sleep 1 2023-08-26T13:54:02.219 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ test_image_replay_state cluster1 mirror image_18 started 2023-08-26T13:54:02.219 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster1 2023-08-26T13:54:02.220 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T13:54:02.220 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=image_18 2023-08-26T13:54:02.220 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local test_state=started 2023-08-26T13:54:02.220 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local status_result 2023-08-26T13:54:02.220 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local current_state=stopped 2023-08-26T13:54:02.222 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_18 2023-08-26T13:54:02.222 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local cluster_instance=cluster1 2023-08-26T13:54:02.222 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ grep -i state 2023-08-26T13:54:02.222 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ shift 2023-08-26T13:54:02.222 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local cluster=cluster1 2023-08-26T13:54:02.223 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local instance=cluster1 2023-08-26T13:54:02.223 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local loop_instance 2023-08-26T13:54:02.223 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-08-26T13:54:02.223 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ sleep 0 2023-08-26T13:54:02.223 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ '[' cluster1 '!=' cluster1 ']' 2023-08-26T13:54:02.224 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ seq 0 3 2023-08-26T13:54:02.226 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-08-26T13:54:02.226 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_18 2023-08-26T13:54:02.227 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local cluster=cluster1:0 2023-08-26T13:54:02.227 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ shift 2023-08-26T13:54:02.227 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local instance 2023-08-26T13:54:02.227 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ set_cluster_instance cluster1:0 cluster instance 2023-08-26T13:54:02.227 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local val=cluster1:0 2023-08-26T13:54:02.228 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local cluster_var_name=cluster 2023-08-26T13:54:02.229 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local instance_var_name=instance 2023-08-26T13:54:02.229 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ cluster=cluster1 2023-08-26T13:54:02.229 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ instance=0 2023-08-26T13:54:02.229 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ '[' 0 = cluster1:0 ']' 2023-08-26T13:54:02.229 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ eval cluster=cluster1 2023-08-26T13:54:02.230 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ cluster=cluster1 2023-08-26T13:54:02.230 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ eval instance=0 2023-08-26T13:54:02.230 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ instance=0 2023-08-26T13:54:02.230 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ daemon_asok_file cluster1:0 cluster1 2023-08-26T13:54:02.230 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ local local_cluster=cluster1:0 2023-08-26T13:54:02.231 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ local cluster=cluster1 2023-08-26T13:54:02.231 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ local instance 2023-08-26T13:54:02.231 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2023-08-26T13:54:02.231 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ local val=cluster1:0 2023-08-26T13:54:02.231 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ local cluster_var_name=local_cluster 2023-08-26T13:54:02.232 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ local instance_var_name=instance 2023-08-26T13:54:02.232 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ cluster=cluster1 2023-08-26T13:54:02.232 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ instance=0 2023-08-26T13:54:02.232 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ '[' 0 = cluster1:0 ']' 2023-08-26T13:54:02.232 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ eval local_cluster=cluster1 2023-08-26T13:54:02.232 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++++ local_cluster=cluster1 2023-08-26T13:54:02.233 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ eval instance=0 2023-08-26T13:54:02.233 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++++ instance=0 2023-08-26T13:54:02.233 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2023-08-26T13:54:02.247 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-08-26T13:54:02.247 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-08-26T13:54:02.247 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-08-26T13:54:02.248 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_18 2023-08-26T13:54:02.356 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:Invalid command: unused arguments: ['mirror/image_18'] 2023-08-26T13:54:02.356 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:rbd mirror status : get status for rbd mirror 2023-08-26T13:54:02.356 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:admin_socket: invalid command 2023-08-26T13:54:02.359 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-08-26T13:54:02.359 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_18 2023-08-26T13:54:02.359 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local cluster=cluster1:1 2023-08-26T13:54:02.359 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ shift 2023-08-26T13:54:02.360 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local instance 2023-08-26T13:54:02.360 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ set_cluster_instance cluster1:1 cluster instance 2023-08-26T13:54:02.360 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local val=cluster1:1 2023-08-26T13:54:02.360 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local cluster_var_name=cluster 2023-08-26T13:54:02.360 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local instance_var_name=instance 2023-08-26T13:54:02.361 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ cluster=cluster1 2023-08-26T13:54:02.361 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ instance=1 2023-08-26T13:54:02.361 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ '[' 1 = cluster1:1 ']' 2023-08-26T13:54:02.361 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ eval cluster=cluster1 2023-08-26T13:54:02.361 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ cluster=cluster1 2023-08-26T13:54:02.361 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ eval instance=1 2023-08-26T13:54:02.362 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ instance=1 2023-08-26T13:54:02.362 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ daemon_asok_file cluster1:1 cluster1 2023-08-26T13:54:02.362 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ local local_cluster=cluster1:1 2023-08-26T13:54:02.362 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ local cluster=cluster1 2023-08-26T13:54:02.362 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ local instance 2023-08-26T13:54:02.363 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2023-08-26T13:54:02.363 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ local val=cluster1:1 2023-08-26T13:54:02.363 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ local cluster_var_name=local_cluster 2023-08-26T13:54:02.363 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ local instance_var_name=instance 2023-08-26T13:54:02.363 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ cluster=cluster1 2023-08-26T13:54:02.364 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ instance=1 2023-08-26T13:54:02.364 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ '[' 1 = cluster1:1 ']' 2023-08-26T13:54:02.364 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ eval local_cluster=cluster1 2023-08-26T13:54:02.364 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++++ local_cluster=cluster1 2023-08-26T13:54:02.364 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ eval instance=1 2023-08-26T13:54:02.364 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++++ instance=1 2023-08-26T13:54:02.365 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2023-08-26T13:54:02.378 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-08-26T13:54:02.378 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-08-26T13:54:02.378 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-08-26T13:54:02.378 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_18 2023-08-26T13:54:02.484 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:Invalid command: unused arguments: ['mirror/image_18'] 2023-08-26T13:54:02.484 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:rbd mirror status : get status for rbd mirror 2023-08-26T13:54:02.484 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:admin_socket: invalid command 2023-08-26T13:54:02.487 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-08-26T13:54:02.488 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/image_18 2023-08-26T13:54:02.488 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local cluster=cluster1:2 2023-08-26T13:54:02.488 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ shift 2023-08-26T13:54:02.488 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local instance 2023-08-26T13:54:02.488 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ set_cluster_instance cluster1:2 cluster instance 2023-08-26T13:54:02.489 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local val=cluster1:2 2023-08-26T13:54:02.489 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local cluster_var_name=cluster 2023-08-26T13:54:02.489 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local instance_var_name=instance 2023-08-26T13:54:02.489 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ cluster=cluster1 2023-08-26T13:54:02.489 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ instance=2 2023-08-26T13:54:02.489 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ '[' 2 = cluster1:2 ']' 2023-08-26T13:54:02.490 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ eval cluster=cluster1 2023-08-26T13:54:02.490 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ cluster=cluster1 2023-08-26T13:54:02.490 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ eval instance=2 2023-08-26T13:54:02.491 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ instance=2 2023-08-26T13:54:02.491 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ daemon_asok_file cluster1:2 cluster1 2023-08-26T13:54:02.491 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ local local_cluster=cluster1:2 2023-08-26T13:54:02.491 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ local cluster=cluster1 2023-08-26T13:54:02.491 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ local instance 2023-08-26T13:54:02.492 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2023-08-26T13:54:02.492 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ local val=cluster1:2 2023-08-26T13:54:02.492 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ local cluster_var_name=local_cluster 2023-08-26T13:54:02.492 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ local instance_var_name=instance 2023-08-26T13:54:02.492 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ cluster=cluster1 2023-08-26T13:54:02.492 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ instance=2 2023-08-26T13:54:02.493 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ '[' 2 = cluster1:2 ']' 2023-08-26T13:54:02.493 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ eval local_cluster=cluster1 2023-08-26T13:54:02.493 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++++ local_cluster=cluster1 2023-08-26T13:54:02.493 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ eval instance=2 2023-08-26T13:54:02.493 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++++ instance=2 2023-08-26T13:54:02.494 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2023-08-26T13:54:02.505 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-08-26T13:54:02.506 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-08-26T13:54:02.506 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-08-26T13:54:02.506 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/image_18 2023-08-26T13:54:02.612 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:Invalid command: unused arguments: ['mirror/image_18'] 2023-08-26T13:54:02.612 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:rbd mirror status : get status for rbd mirror 2023-08-26T13:54:02.612 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:admin_socket: invalid command 2023-08-26T13:54:02.616 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-08-26T13:54:02.616 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ admin_daemon cluster1:3 rbd mirror status mirror/image_18 2023-08-26T13:54:02.617 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local cluster=cluster1:3 2023-08-26T13:54:02.617 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ shift 2023-08-26T13:54:02.617 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local instance 2023-08-26T13:54:02.617 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ set_cluster_instance cluster1:3 cluster instance 2023-08-26T13:54:02.617 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local val=cluster1:3 2023-08-26T13:54:02.618 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local cluster_var_name=cluster 2023-08-26T13:54:02.618 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local instance_var_name=instance 2023-08-26T13:54:02.618 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ cluster=cluster1 2023-08-26T13:54:02.618 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ instance=3 2023-08-26T13:54:02.618 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ '[' 3 = cluster1:3 ']' 2023-08-26T13:54:02.619 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ eval cluster=cluster1 2023-08-26T13:54:02.619 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ cluster=cluster1 2023-08-26T13:54:02.619 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ eval instance=3 2023-08-26T13:54:02.619 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ instance=3 2023-08-26T13:54:02.619 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ daemon_asok_file cluster1:3 cluster1 2023-08-26T13:54:02.620 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ local local_cluster=cluster1:3 2023-08-26T13:54:02.620 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ local cluster=cluster1 2023-08-26T13:54:02.620 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ local instance 2023-08-26T13:54:02.620 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ set_cluster_instance cluster1:3 local_cluster instance 2023-08-26T13:54:02.620 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ local val=cluster1:3 2023-08-26T13:54:02.621 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ local cluster_var_name=local_cluster 2023-08-26T13:54:02.621 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ local instance_var_name=instance 2023-08-26T13:54:02.621 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ cluster=cluster1 2023-08-26T13:54:02.621 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ instance=3 2023-08-26T13:54:02.621 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ '[' 3 = cluster1:3 ']' 2023-08-26T13:54:02.621 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ eval local_cluster=cluster1 2023-08-26T13:54:02.622 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++++ local_cluster=cluster1 2023-08-26T13:54:02.622 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ eval instance=3 2023-08-26T13:54:02.622 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++++ instance=3 2023-08-26T13:54:02.622 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2023-08-26T13:54:02.634 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2023-08-26T13:54:02.635 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2023-08-26T13:54:02.635 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2023-08-26T13:54:02.635 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror status mirror/image_18 2023-08-26T13:54:02.736 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.0 with skip (0.3928852084183757) > weight (0.25) 2023-08-26T13:54:02.736 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.1 with skip (0.4831080847429289) > weight (0.25) 2023-08-26T13:54:02.737 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.2 with skip (0.5530985150839761) > weight (0.25) 2023-08-26T13:54:02.737 INFO:tasks.rbd_mirror_thrash:kill cluster1.client.mirror.3 2023-08-26T13:54:02.737 INFO:tasks.rbd_mirror.cluster1.client.mirror.3:Sent signal 15 2023-08-26T13:54:02.737 INFO:tasks.rbd_mirror_thrash:waiting for 7 secs before reviving daemons 2023-08-26T13:54:02.738 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi124.stderr:2023-08-26T13:54:02.736+0000 7fa1993cc700 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.3 (PID: 106012) UID: 1000 2023-08-26T13:54:02.754 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ return 0 2023-08-26T13:54:02.755 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ status_result=' "state": "Replaying"' 2023-08-26T13:54:02.755 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ echo ' "state": "Replaying"' 2023-08-26T13:54:02.755 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ grep -i Replaying 2023-08-26T13:54:02.755 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: "state": "Replaying" 2023-08-26T13:54:02.758 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ current_state=started 2023-08-26T13:54:02.758 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ test started = started 2023-08-26T13:54:02.758 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ return 0 2023-08-26T13:54:02.758 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_18 2023-08-26T13:54:02.758 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local local_cluster=cluster1 2023-08-26T13:54:02.759 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster2 2023-08-26T13:54:02.759 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T13:54:02.759 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=image_18 2023-08-26T13:54:02.759 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ '[' snapshot = journal ']' 2023-08-26T13:54:02.759 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ '[' snapshot = snapshot ']' 2023-08-26T13:54:02.760 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_18 2023-08-26T13:54:02.760 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local local_cluster=cluster1 2023-08-26T13:54:02.760 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster2 2023-08-26T13:54:02.760 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T13:54:02.760 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=image_18 2023-08-26T13:54:02.760 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ mkfname cluster2-mirror-image_18.status 2023-08-26T13:54:02.761 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ echo cluster2-mirror-image_18.status 2023-08-26T13:54:02.761 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ sed -e 's|[/ ]|_|g' 2023-08-26T13:54:02.761 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local status_log=/tmp/tmp.TfDn6BcQ3N/cluster2-mirror-image_18.status 2023-08-26T13:54:02.761 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ mkfname cluster1-mirror-image_18.status 2023-08-26T13:54:02.761 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ echo cluster1-mirror-image_18.status 2023-08-26T13:54:02.762 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ sed -e 's|[/ ]|_|g' 2023-08-26T13:54:02.762 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local local_status_log=/tmp/tmp.TfDn6BcQ3N/cluster1-mirror-image_18.status 2023-08-26T13:54:02.763 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ mirror_image_snapshot cluster2 mirror image_18 2023-08-26T13:54:02.763 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster2 2023-08-26T13:54:02.763 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T13:54:02.763 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=image_18 2023-08-26T13:54:02.763 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_18 2023-08-26T13:54:03.223 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:Snapshot ID: 153 2023-08-26T13:54:03.231 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_18 /tmp/tmp.TfDn6BcQ3N/cluster2-mirror-image_18.status 2023-08-26T13:54:03.231 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster2 2023-08-26T13:54:03.231 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T13:54:03.231 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=image_18 2023-08-26T13:54:03.232 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local log=/tmp/tmp.TfDn6BcQ3N/cluster2-mirror-image_18.status 2023-08-26T13:54:03.232 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_18 --format xml 2023-08-26T13:54:03.232 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-08-26T13:54:03.274 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ xmlstarlet sel -t -v //snapshot/id 2023-08-26T13:54:03.276 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local snapshot_id=153 2023-08-26T13:54:03.276 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ true 2023-08-26T13:54:03.276 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-08-26T13:54:03.276 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ sleep 0.2 2023-08-26T13:54:03.477 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_18 /tmp/tmp.TfDn6BcQ3N/cluster1-mirror-image_18.status 2023-08-26T13:54:03.477 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster1 2023-08-26T13:54:03.477 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T13:54:03.478 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=image_18 2023-08-26T13:54:03.478 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local log=/tmp/tmp.TfDn6BcQ3N/cluster1-mirror-image_18.status 2023-08-26T13:54:03.478 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_18 --format xml 2023-08-26T13:54:03.478 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-08-26T13:54:03.524 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-08-26T13:54:03.525 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local primary_snapshot_id=85 2023-08-26T13:54:03.526 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ test 153 = 85 2023-08-26T13:54:03.526 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-08-26T13:54:03.526 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ sleep 0.4 2023-08-26T13:54:03.927 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_18 /tmp/tmp.TfDn6BcQ3N/cluster1-mirror-image_18.status 2023-08-26T13:54:03.927 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster1 2023-08-26T13:54:03.928 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T13:54:03.928 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=image_18 2023-08-26T13:54:03.928 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local log=/tmp/tmp.TfDn6BcQ3N/cluster1-mirror-image_18.status 2023-08-26T13:54:03.928 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-08-26T13:54:03.929 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_18 --format xml 2023-08-26T13:54:04.031 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-08-26T13:54:04.033 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local primary_snapshot_id=85 2023-08-26T13:54:04.034 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ test 153 = 85 2023-08-26T13:54:04.034 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-08-26T13:54:04.034 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ sleep 0.8 2023-08-26T13:54:04.835 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_18 /tmp/tmp.TfDn6BcQ3N/cluster1-mirror-image_18.status 2023-08-26T13:54:04.836 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster1 2023-08-26T13:54:04.836 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T13:54:04.837 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=image_18 2023-08-26T13:54:04.837 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local log=/tmp/tmp.TfDn6BcQ3N/cluster1-mirror-image_18.status 2023-08-26T13:54:04.837 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_18 --format xml 2023-08-26T13:54:04.837 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-08-26T13:54:04.882 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-08-26T13:54:04.883 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local primary_snapshot_id=85 2023-08-26T13:54:04.883 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ test 153 = 85 2023-08-26T13:54:04.884 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-08-26T13:54:04.884 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ sleep 1.6 2023-08-26T13:54:06.485 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_18 /tmp/tmp.TfDn6BcQ3N/cluster1-mirror-image_18.status 2023-08-26T13:54:06.485 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster1 2023-08-26T13:54:06.485 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T13:54:06.486 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=image_18 2023-08-26T13:54:06.486 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local log=/tmp/tmp.TfDn6BcQ3N/cluster1-mirror-image_18.status 2023-08-26T13:54:06.486 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_18 --format xml 2023-08-26T13:54:06.486 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-08-26T13:54:06.536 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-08-26T13:54:06.538 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local primary_snapshot_id=85 2023-08-26T13:54:06.538 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ test 153 = 85 2023-08-26T13:54:06.539 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-08-26T13:54:06.539 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ sleep 2 2023-08-26T13:54:08.540 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_18 /tmp/tmp.TfDn6BcQ3N/cluster1-mirror-image_18.status 2023-08-26T13:54:08.540 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster1 2023-08-26T13:54:08.540 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T13:54:08.540 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=image_18 2023-08-26T13:54:08.541 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local log=/tmp/tmp.TfDn6BcQ3N/cluster1-mirror-image_18.status 2023-08-26T13:54:08.541 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_18 --format xml 2023-08-26T13:54:08.541 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-08-26T13:54:08.591 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-08-26T13:54:08.594 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local primary_snapshot_id=85 2023-08-26T13:54:08.594 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ test 153 = 85 2023-08-26T13:54:08.594 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-08-26T13:54:08.594 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ sleep 2 2023-08-26T13:54:09.739 INFO:tasks.rbd_mirror_thrash:waiting for cluster1.client.mirror.3 2023-08-26T13:54:09.739 INFO:teuthology.orchestra.run:waiting for 600 2023-08-26T13:54:09.739 INFO:tasks.rbd_mirror_thrash:reviving cluster1.client.mirror.3 2023-08-26T13:54:09.740 INFO:tasks.rbd_mirror.cluster1.client.mirror.3:Restarting daemon 2023-08-26T13:54:09.740 DEBUG:teuthology.orchestra.run.smithi124:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.3 2023-08-26T13:54:09.743 INFO:tasks.rbd_mirror.cluster1.client.mirror.3:Started 2023-08-26T13:54:09.743 INFO:tasks.rbd_mirror_thrash:waiting for 63 secs before thrashing 2023-08-26T13:54:10.595 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_18 /tmp/tmp.TfDn6BcQ3N/cluster1-mirror-image_18.status 2023-08-26T13:54:10.596 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster1 2023-08-26T13:54:10.596 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T13:54:10.596 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=image_18 2023-08-26T13:54:10.596 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local log=/tmp/tmp.TfDn6BcQ3N/cluster1-mirror-image_18.status 2023-08-26T13:54:10.597 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_18 --format xml 2023-08-26T13:54:10.597 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-08-26T13:54:10.643 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-08-26T13:54:10.645 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local primary_snapshot_id=85 2023-08-26T13:54:10.646 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ test 153 = 85 2023-08-26T13:54:10.646 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-08-26T13:54:10.646 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ sleep 4 2023-08-26T13:54:14.646 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_18 /tmp/tmp.TfDn6BcQ3N/cluster1-mirror-image_18.status 2023-08-26T13:54:14.647 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster1 2023-08-26T13:54:14.647 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T13:54:14.647 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=image_18 2023-08-26T13:54:14.647 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local log=/tmp/tmp.TfDn6BcQ3N/cluster1-mirror-image_18.status 2023-08-26T13:54:14.647 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_18 --format xml 2023-08-26T13:54:14.648 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-08-26T13:54:14.695 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-08-26T13:54:14.697 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local primary_snapshot_id=85 2023-08-26T13:54:14.697 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ test 153 = 85 2023-08-26T13:54:14.697 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-08-26T13:54:14.698 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ sleep 4 2023-08-26T13:54:18.698 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_18 /tmp/tmp.TfDn6BcQ3N/cluster1-mirror-image_18.status 2023-08-26T13:54:18.699 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster1 2023-08-26T13:54:18.699 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T13:54:18.699 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=image_18 2023-08-26T13:54:18.699 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local log=/tmp/tmp.TfDn6BcQ3N/cluster1-mirror-image_18.status 2023-08-26T13:54:18.700 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_18 --format xml 2023-08-26T13:54:18.700 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-08-26T13:54:18.751 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-08-26T13:54:18.753 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local primary_snapshot_id=85 2023-08-26T13:54:18.753 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ test 153 = 85 2023-08-26T13:54:18.753 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-08-26T13:54:18.753 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ sleep 8 2023-08-26T13:54:19.521 INFO:tasks.rbd_mirror_thrash:kill cluster2.client.mirror.0 2023-08-26T13:54:19.521 INFO:tasks.rbd_mirror.cluster2.client.mirror.0:Sent signal 15 2023-08-26T13:54:19.522 INFO:tasks.rbd_mirror_thrash:waiting for 6 secs before reviving daemons 2023-08-26T13:54:19.522 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi124.stderr:2023-08-26T13:54:19.521+0000 7fec522d5700 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.0 (PID: 115050) UID: 1000 2023-08-26T13:54:25.523 INFO:tasks.rbd_mirror_thrash:waiting for cluster2.client.mirror.0 2023-08-26T13:54:25.523 INFO:teuthology.orchestra.run:waiting for 600 2023-08-26T13:54:25.524 INFO:tasks.rbd_mirror_thrash:reviving cluster2.client.mirror.0 2023-08-26T13:54:25.524 INFO:tasks.rbd_mirror.cluster2.client.mirror.0:Restarting daemon 2023-08-26T13:54:25.524 DEBUG:teuthology.orchestra.run.smithi124:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.0 2023-08-26T13:54:25.528 INFO:tasks.rbd_mirror.cluster2.client.mirror.0:Started 2023-08-26T13:54:25.528 INFO:tasks.rbd_mirror_thrash:waiting for 118 secs before thrashing 2023-08-26T13:54:26.754 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_18 /tmp/tmp.TfDn6BcQ3N/cluster1-mirror-image_18.status 2023-08-26T13:54:26.755 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster1 2023-08-26T13:54:26.755 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T13:54:26.755 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=image_18 2023-08-26T13:54:26.755 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local log=/tmp/tmp.TfDn6BcQ3N/cluster1-mirror-image_18.status 2023-08-26T13:54:26.756 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_18 --format xml 2023-08-26T13:54:26.756 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-08-26T13:54:26.805 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-08-26T13:54:26.807 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local primary_snapshot_id=85 2023-08-26T13:54:26.807 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ test 153 = 85 2023-08-26T13:54:26.807 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-08-26T13:54:26.807 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ sleep 8 2023-08-26T13:54:34.808 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_18 /tmp/tmp.TfDn6BcQ3N/cluster1-mirror-image_18.status 2023-08-26T13:54:34.809 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster1 2023-08-26T13:54:34.809 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T13:54:34.809 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=image_18 2023-08-26T13:54:34.809 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local log=/tmp/tmp.TfDn6BcQ3N/cluster1-mirror-image_18.status 2023-08-26T13:54:34.809 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_18 --format xml 2023-08-26T13:54:34.810 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-08-26T13:54:34.858 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-08-26T13:54:34.860 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local primary_snapshot_id=85 2023-08-26T13:54:34.860 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ test 153 = 85 2023-08-26T13:54:34.861 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-08-26T13:54:34.861 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ sleep 16 2023-08-26T13:54:50.862 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_18 /tmp/tmp.TfDn6BcQ3N/cluster1-mirror-image_18.status 2023-08-26T13:54:50.862 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster1 2023-08-26T13:54:50.862 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T13:54:50.863 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=image_18 2023-08-26T13:54:50.863 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local log=/tmp/tmp.TfDn6BcQ3N/cluster1-mirror-image_18.status 2023-08-26T13:54:50.863 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_18 --format xml 2023-08-26T13:54:50.863 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-08-26T13:54:50.913 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-08-26T13:54:50.915 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local primary_snapshot_id=153 2023-08-26T13:54:50.916 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ test 153 = 153 2023-08-26T13:54:50.916 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ return 0 2023-08-26T13:54:50.916 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ wait_for_snap_present cluster1 mirror image_18 snap 2023-08-26T13:54:50.916 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster1 2023-08-26T13:54:50.916 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T13:54:50.917 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=image_18 2023-08-26T13:54:50.917 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local snap_name=snap 2023-08-26T13:54:50.917 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local s 2023-08-26T13:54:50.917 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2023-08-26T13:54:50.917 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ sleep 1 2023-08-26T13:54:51.916 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rbd --cluster cluster1 info mirror/image_18@snap 2023-08-26T13:54:51.961 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:rbd image 'image_18': 2023-08-26T13:54:51.962 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: size 128 MiB in 32 objects 2023-08-26T13:54:51.962 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: order 22 (4 MiB objects) 2023-08-26T13:54:51.962 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: snapshot_count: 2 2023-08-26T13:54:51.962 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: id: 11d295cd736a 2023-08-26T13:54:51.963 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: block_name_prefix: rbd_data.11d295cd736a 2023-08-26T13:54:51.963 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: format: 2 2023-08-26T13:54:51.963 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: features: layering, exclusive-lock, non-primary 2023-08-26T13:54:51.963 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: op_features: 2023-08-26T13:54:51.963 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: flags: 2023-08-26T13:54:51.964 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: create_timestamp: Sat Aug 26 13:45:39 2023 2023-08-26T13:54:51.964 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: access_timestamp: Sat Aug 26 13:45:39 2023 2023-08-26T13:54:51.964 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: modify_timestamp: Sat Aug 26 13:45:39 2023 2023-08-26T13:54:51.964 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: protected: False 2023-08-26T13:54:51.965 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: mirroring state: enabled 2023-08-26T13:54:51.965 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: mirroring mode: snapshot 2023-08-26T13:54:51.965 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: mirroring global id: 5293326c-bb2a-4d36-bfa9-7cfd4da48fe1 2023-08-26T13:54:51.965 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: mirroring primary: false 2023-08-26T13:54:51.966 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ return 0 2023-08-26T13:54:51.967 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ compare_image_snaps mirror image_18 snap 2023-08-26T13:54:51.967 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T13:54:51.967 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=image_18 2023-08-26T13:54:51.967 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local snap_name=snap 2023-08-26T13:54:51.967 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local ret=0 2023-08-26T13:54:51.968 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local rmt_export=/tmp/tmp.TfDn6BcQ3N/cluster2-mirror-image_18.export 2023-08-26T13:54:51.968 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local loc_export=/tmp/tmp.TfDn6BcQ3N/cluster1-mirror-image_18.export 2023-08-26T13:54:51.968 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rm -f /tmp/tmp.TfDn6BcQ3N/cluster2-mirror-image_18.export /tmp/tmp.TfDn6BcQ3N/cluster1-mirror-image_18.export 2023-08-26T13:54:51.968 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rbd --cluster cluster2 -p mirror export image_18@snap /tmp/tmp.TfDn6BcQ3N/cluster2-mirror-image_18.export 2023-08-26T13:54:52.145 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-08-26T13:54:52.151 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rbd --cluster cluster1 -p mirror export image_18@snap /tmp/tmp.TfDn6BcQ3N/cluster1-mirror-image_18.export 2023-08-26T13:54:52.350 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-08-26T13:54:52.357 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ cmp /tmp/tmp.TfDn6BcQ3N/cluster2-mirror-image_18.export /tmp/tmp.TfDn6BcQ3N/cluster1-mirror-image_18.export 2023-08-26T13:54:52.411 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rm -f /tmp/tmp.TfDn6BcQ3N/cluster2-mirror-image_18.export /tmp/tmp.TfDn6BcQ3N/cluster1-mirror-image_18.export 2023-08-26T13:54:52.448 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ return 0 2023-08-26T13:54:52.449 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-08-26T13:54:52.449 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ image=image_19 2023-08-26T13:54:52.449 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ create_snap cluster2 mirror image_19 snap 2023-08-26T13:54:52.449 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster2 2023-08-26T13:54:52.449 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T13:54:52.450 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=image_19 2023-08-26T13:54:52.450 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local snap_name=snap 2023-08-26T13:54:52.450 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rbd --cluster cluster2 -p mirror snap create image_19@snap --debug-rbd=20 --debug-journaler=20 2023-08-26T13:54:54.250 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ wait_for_image_replay_started cluster1 mirror image_19 2023-08-26T13:54:54.251 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster1 2023-08-26T13:54:54.251 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T13:54:54.251 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=image_19 2023-08-26T13:54:54.251 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ wait_for_image_replay_state cluster1 mirror image_19 started 2023-08-26T13:54:54.251 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster1 2023-08-26T13:54:54.252 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T13:54:54.252 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=image_19 2023-08-26T13:54:54.252 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local state=started 2023-08-26T13:54:54.252 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local s 2023-08-26T13:54:54.252 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-08-26T13:54:54.253 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ sleep 1 2023-08-26T13:54:55.251 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ test_image_replay_state cluster1 mirror image_19 started 2023-08-26T13:54:55.252 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster1 2023-08-26T13:54:55.252 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T13:54:55.252 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=image_19 2023-08-26T13:54:55.252 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local test_state=started 2023-08-26T13:54:55.253 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local status_result 2023-08-26T13:54:55.253 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local current_state=stopped 2023-08-26T13:54:55.254 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_19 2023-08-26T13:54:55.254 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local cluster_instance=cluster1 2023-08-26T13:54:55.254 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ shift 2023-08-26T13:54:55.255 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ grep -i state 2023-08-26T13:54:55.255 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local cluster=cluster1 2023-08-26T13:54:55.255 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local instance=cluster1 2023-08-26T13:54:55.255 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local loop_instance 2023-08-26T13:54:55.255 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-08-26T13:54:55.256 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ sleep 0 2023-08-26T13:54:55.256 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ '[' cluster1 '!=' cluster1 ']' 2023-08-26T13:54:55.256 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ seq 0 3 2023-08-26T13:54:55.258 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-08-26T13:54:55.258 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_19 2023-08-26T13:54:55.258 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local cluster=cluster1:0 2023-08-26T13:54:55.258 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ shift 2023-08-26T13:54:55.258 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local instance 2023-08-26T13:54:55.259 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ set_cluster_instance cluster1:0 cluster instance 2023-08-26T13:54:55.259 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local val=cluster1:0 2023-08-26T13:54:55.259 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local cluster_var_name=cluster 2023-08-26T13:54:55.259 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local instance_var_name=instance 2023-08-26T13:54:55.259 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ cluster=cluster1 2023-08-26T13:54:55.260 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ instance=0 2023-08-26T13:54:55.260 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ '[' 0 = cluster1:0 ']' 2023-08-26T13:54:55.260 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ eval cluster=cluster1 2023-08-26T13:54:55.261 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ cluster=cluster1 2023-08-26T13:54:55.261 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ eval instance=0 2023-08-26T13:54:55.261 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ instance=0 2023-08-26T13:54:55.261 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ daemon_asok_file cluster1:0 cluster1 2023-08-26T13:54:55.263 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ local local_cluster=cluster1:0 2023-08-26T13:54:55.263 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ local cluster=cluster1 2023-08-26T13:54:55.263 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ local instance 2023-08-26T13:54:55.263 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2023-08-26T13:54:55.264 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ local val=cluster1:0 2023-08-26T13:54:55.264 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ local cluster_var_name=local_cluster 2023-08-26T13:54:55.264 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ local instance_var_name=instance 2023-08-26T13:54:55.264 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ cluster=cluster1 2023-08-26T13:54:55.264 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ instance=0 2023-08-26T13:54:55.265 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ '[' 0 = cluster1:0 ']' 2023-08-26T13:54:55.265 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ eval local_cluster=cluster1 2023-08-26T13:54:55.265 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++++ local_cluster=cluster1 2023-08-26T13:54:55.265 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ eval instance=0 2023-08-26T13:54:55.265 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++++ instance=0 2023-08-26T13:54:55.265 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2023-08-26T13:54:55.278 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-08-26T13:54:55.278 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-08-26T13:54:55.279 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-08-26T13:54:55.279 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_19 2023-08-26T13:54:55.396 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:Invalid command: unused arguments: ['mirror/image_19'] 2023-08-26T13:54:55.396 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:rbd mirror status : get status for rbd mirror 2023-08-26T13:54:55.397 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:admin_socket: invalid command 2023-08-26T13:54:55.400 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-08-26T13:54:55.400 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_19 2023-08-26T13:54:55.400 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local cluster=cluster1:1 2023-08-26T13:54:55.400 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ shift 2023-08-26T13:54:55.400 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local instance 2023-08-26T13:54:55.401 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ set_cluster_instance cluster1:1 cluster instance 2023-08-26T13:54:55.401 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local val=cluster1:1 2023-08-26T13:54:55.401 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local cluster_var_name=cluster 2023-08-26T13:54:55.401 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local instance_var_name=instance 2023-08-26T13:54:55.401 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ cluster=cluster1 2023-08-26T13:54:55.401 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ instance=1 2023-08-26T13:54:55.402 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ '[' 1 = cluster1:1 ']' 2023-08-26T13:54:55.402 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ eval cluster=cluster1 2023-08-26T13:54:55.402 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ cluster=cluster1 2023-08-26T13:54:55.402 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ eval instance=1 2023-08-26T13:54:55.402 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ instance=1 2023-08-26T13:54:55.403 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ daemon_asok_file cluster1:1 cluster1 2023-08-26T13:54:55.403 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ local local_cluster=cluster1:1 2023-08-26T13:54:55.403 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ local cluster=cluster1 2023-08-26T13:54:55.403 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ local instance 2023-08-26T13:54:55.403 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2023-08-26T13:54:55.404 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ local val=cluster1:1 2023-08-26T13:54:55.404 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ local cluster_var_name=local_cluster 2023-08-26T13:54:55.404 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ local instance_var_name=instance 2023-08-26T13:54:55.404 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ cluster=cluster1 2023-08-26T13:54:55.404 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ instance=1 2023-08-26T13:54:55.405 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ '[' 1 = cluster1:1 ']' 2023-08-26T13:54:55.405 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ eval local_cluster=cluster1 2023-08-26T13:54:55.405 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++++ local_cluster=cluster1 2023-08-26T13:54:55.405 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ eval instance=1 2023-08-26T13:54:55.405 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++++ instance=1 2023-08-26T13:54:55.406 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2023-08-26T13:54:55.418 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-08-26T13:54:55.418 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-08-26T13:54:55.419 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-08-26T13:54:55.419 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_19 2023-08-26T13:54:55.532 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ return 0 2023-08-26T13:54:55.533 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ status_result=' "state": "Replaying"' 2023-08-26T13:54:55.533 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ echo ' "state": "Replaying"' 2023-08-26T13:54:55.533 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ grep -i Replaying 2023-08-26T13:54:55.534 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: "state": "Replaying" 2023-08-26T13:54:55.536 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ current_state=started 2023-08-26T13:54:55.536 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ test started = started 2023-08-26T13:54:55.536 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ return 0 2023-08-26T13:54:55.537 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_19 2023-08-26T13:54:55.537 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local local_cluster=cluster1 2023-08-26T13:54:55.537 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster2 2023-08-26T13:54:55.537 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T13:54:55.537 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=image_19 2023-08-26T13:54:55.537 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ '[' snapshot = journal ']' 2023-08-26T13:54:55.538 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ '[' snapshot = snapshot ']' 2023-08-26T13:54:55.538 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_19 2023-08-26T13:54:55.538 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local local_cluster=cluster1 2023-08-26T13:54:55.538 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster2 2023-08-26T13:54:55.538 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T13:54:55.539 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=image_19 2023-08-26T13:54:55.539 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ mkfname cluster2-mirror-image_19.status 2023-08-26T13:54:55.539 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ echo cluster2-mirror-image_19.status 2023-08-26T13:54:55.539 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ sed -e 's|[/ ]|_|g' 2023-08-26T13:54:55.539 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local status_log=/tmp/tmp.TfDn6BcQ3N/cluster2-mirror-image_19.status 2023-08-26T13:54:55.539 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ mkfname cluster1-mirror-image_19.status 2023-08-26T13:54:55.540 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ echo cluster1-mirror-image_19.status 2023-08-26T13:54:55.540 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ sed -e 's|[/ ]|_|g' 2023-08-26T13:54:55.541 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local local_status_log=/tmp/tmp.TfDn6BcQ3N/cluster1-mirror-image_19.status 2023-08-26T13:54:55.541 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ mirror_image_snapshot cluster2 mirror image_19 2023-08-26T13:54:55.541 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster2 2023-08-26T13:54:55.542 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T13:54:55.542 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=image_19 2023-08-26T13:54:55.542 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_19 2023-08-26T13:54:56.262 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:Snapshot ID: 155 2023-08-26T13:54:56.271 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_19 /tmp/tmp.TfDn6BcQ3N/cluster2-mirror-image_19.status 2023-08-26T13:54:56.272 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster2 2023-08-26T13:54:56.272 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T13:54:56.272 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=image_19 2023-08-26T13:54:56.272 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local log=/tmp/tmp.TfDn6BcQ3N/cluster2-mirror-image_19.status 2023-08-26T13:54:56.273 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_19 --format xml 2023-08-26T13:54:56.273 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-08-26T13:54:56.316 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ xmlstarlet sel -t -v //snapshot/id 2023-08-26T13:54:56.318 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local snapshot_id=155 2023-08-26T13:54:56.318 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ true 2023-08-26T13:54:56.319 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-08-26T13:54:56.319 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ sleep 0.2 2023-08-26T13:54:56.519 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_19 /tmp/tmp.TfDn6BcQ3N/cluster1-mirror-image_19.status 2023-08-26T13:54:56.520 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster1 2023-08-26T13:54:56.520 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T13:54:56.520 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=image_19 2023-08-26T13:54:56.520 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local log=/tmp/tmp.TfDn6BcQ3N/cluster1-mirror-image_19.status 2023-08-26T13:54:56.521 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_19 --format xml 2023-08-26T13:54:56.521 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-08-26T13:54:56.566 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ true 2023-08-26T13:54:56.566 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-08-26T13:54:56.568 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:-:1.1: Document is empty 2023-08-26T13:54:56.568 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr: 2023-08-26T13:54:56.569 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:^ 2023-08-26T13:54:56.569 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local primary_snapshot_id= 2023-08-26T13:54:56.569 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ test 155 = '' 2023-08-26T13:54:56.570 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-08-26T13:54:56.570 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ sleep 0.4 2023-08-26T13:54:56.969 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_19 /tmp/tmp.TfDn6BcQ3N/cluster1-mirror-image_19.status 2023-08-26T13:54:56.970 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster1 2023-08-26T13:54:56.970 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T13:54:56.970 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=image_19 2023-08-26T13:54:56.970 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local log=/tmp/tmp.TfDn6BcQ3N/cluster1-mirror-image_19.status 2023-08-26T13:54:56.970 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_19 --format xml 2023-08-26T13:54:56.971 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-08-26T13:54:57.017 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ true 2023-08-26T13:54:57.017 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-08-26T13:54:57.019 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:-:1.1: Document is empty 2023-08-26T13:54:57.019 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr: 2023-08-26T13:54:57.019 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:^ 2023-08-26T13:54:57.020 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local primary_snapshot_id= 2023-08-26T13:54:57.020 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ test 155 = '' 2023-08-26T13:54:57.020 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-08-26T13:54:57.020 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ sleep 0.8 2023-08-26T13:54:57.821 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_19 /tmp/tmp.TfDn6BcQ3N/cluster1-mirror-image_19.status 2023-08-26T13:54:57.821 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster1 2023-08-26T13:54:57.821 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T13:54:57.821 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=image_19 2023-08-26T13:54:57.822 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local log=/tmp/tmp.TfDn6BcQ3N/cluster1-mirror-image_19.status 2023-08-26T13:54:57.822 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_19 --format xml 2023-08-26T13:54:57.822 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-08-26T13:54:57.865 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-08-26T13:54:57.867 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local primary_snapshot_id=155 2023-08-26T13:54:57.868 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ test 155 = 155 2023-08-26T13:54:57.868 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ return 0 2023-08-26T13:54:57.868 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ wait_for_snap_present cluster1 mirror image_19 snap 2023-08-26T13:54:57.868 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster1 2023-08-26T13:54:57.868 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T13:54:57.869 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=image_19 2023-08-26T13:54:57.869 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local snap_name=snap 2023-08-26T13:54:57.869 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local s 2023-08-26T13:54:57.869 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2023-08-26T13:54:57.869 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ sleep 1 2023-08-26T13:54:58.868 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rbd --cluster cluster1 info mirror/image_19@snap 2023-08-26T13:54:58.914 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:rbd image 'image_19': 2023-08-26T13:54:58.914 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: size 128 MiB in 32 objects 2023-08-26T13:54:58.915 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: order 22 (4 MiB objects) 2023-08-26T13:54:58.915 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: snapshot_count: 2 2023-08-26T13:54:58.915 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: id: 122bc0cf705b 2023-08-26T13:54:58.915 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: block_name_prefix: rbd_data.122bc0cf705b 2023-08-26T13:54:58.915 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: format: 2 2023-08-26T13:54:58.916 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: features: layering, exclusive-lock, non-primary 2023-08-26T13:54:58.916 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: op_features: 2023-08-26T13:54:58.916 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: flags: 2023-08-26T13:54:58.916 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: create_timestamp: Sat Aug 26 13:45:45 2023 2023-08-26T13:54:58.916 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: access_timestamp: Sat Aug 26 13:45:45 2023 2023-08-26T13:54:58.916 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: modify_timestamp: Sat Aug 26 13:45:45 2023 2023-08-26T13:54:58.917 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: protected: False 2023-08-26T13:54:58.917 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: mirroring state: enabled 2023-08-26T13:54:58.917 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: mirroring mode: snapshot 2023-08-26T13:54:58.917 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: mirroring global id: f1b3d4dd-7b2d-44de-a0df-388999259524 2023-08-26T13:54:58.917 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: mirroring primary: false 2023-08-26T13:54:58.919 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ return 0 2023-08-26T13:54:58.919 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ compare_image_snaps mirror image_19 snap 2023-08-26T13:54:58.919 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T13:54:58.919 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=image_19 2023-08-26T13:54:58.919 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local snap_name=snap 2023-08-26T13:54:58.920 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local ret=0 2023-08-26T13:54:58.920 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local rmt_export=/tmp/tmp.TfDn6BcQ3N/cluster2-mirror-image_19.export 2023-08-26T13:54:58.920 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local loc_export=/tmp/tmp.TfDn6BcQ3N/cluster1-mirror-image_19.export 2023-08-26T13:54:58.920 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rm -f /tmp/tmp.TfDn6BcQ3N/cluster2-mirror-image_19.export /tmp/tmp.TfDn6BcQ3N/cluster1-mirror-image_19.export 2023-08-26T13:54:58.920 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rbd --cluster cluster2 -p mirror export image_19@snap /tmp/tmp.TfDn6BcQ3N/cluster2-mirror-image_19.export 2023-08-26T13:54:59.103 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-08-26T13:54:59.108 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rbd --cluster cluster1 -p mirror export image_19@snap /tmp/tmp.TfDn6BcQ3N/cluster1-mirror-image_19.export 2023-08-26T13:54:59.347 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-08-26T13:54:59.353 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ cmp /tmp/tmp.TfDn6BcQ3N/cluster2-mirror-image_19.export /tmp/tmp.TfDn6BcQ3N/cluster1-mirror-image_19.export 2023-08-26T13:54:59.408 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rm -f /tmp/tmp.TfDn6BcQ3N/cluster2-mirror-image_19.export /tmp/tmp.TfDn6BcQ3N/cluster1-mirror-image_19.export 2023-08-26T13:54:59.444 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ return 0 2023-08-26T13:54:59.445 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-08-26T13:54:59.445 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ image=image_20 2023-08-26T13:54:59.445 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ create_snap cluster2 mirror image_20 snap 2023-08-26T13:54:59.445 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster2 2023-08-26T13:54:59.445 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T13:54:59.446 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=image_20 2023-08-26T13:54:59.446 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local snap_name=snap 2023-08-26T13:54:59.446 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rbd --cluster cluster2 -p mirror snap create image_20@snap --debug-rbd=20 --debug-journaler=20 2023-08-26T13:55:01.269 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ wait_for_image_replay_started cluster1 mirror image_20 2023-08-26T13:55:01.270 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster1 2023-08-26T13:55:01.270 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T13:55:01.270 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=image_20 2023-08-26T13:55:01.270 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ wait_for_image_replay_state cluster1 mirror image_20 started 2023-08-26T13:55:01.270 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster1 2023-08-26T13:55:01.271 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T13:55:01.271 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=image_20 2023-08-26T13:55:01.271 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local state=started 2023-08-26T13:55:01.271 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local s 2023-08-26T13:55:01.271 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-08-26T13:55:01.272 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ sleep 1 2023-08-26T13:55:02.271 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ test_image_replay_state cluster1 mirror image_20 started 2023-08-26T13:55:02.271 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster1 2023-08-26T13:55:02.272 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T13:55:02.272 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=image_20 2023-08-26T13:55:02.272 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local test_state=started 2023-08-26T13:55:02.272 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local status_result 2023-08-26T13:55:02.272 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local current_state=stopped 2023-08-26T13:55:02.273 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_20 2023-08-26T13:55:02.273 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local cluster_instance=cluster1 2023-08-26T13:55:02.273 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ grep -i state 2023-08-26T13:55:02.273 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ shift 2023-08-26T13:55:02.273 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local cluster=cluster1 2023-08-26T13:55:02.273 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local instance=cluster1 2023-08-26T13:55:02.274 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local loop_instance 2023-08-26T13:55:02.274 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-08-26T13:55:02.274 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ sleep 0 2023-08-26T13:55:02.276 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ '[' cluster1 '!=' cluster1 ']' 2023-08-26T13:55:02.276 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ seq 0 3 2023-08-26T13:55:02.276 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-08-26T13:55:02.276 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_20 2023-08-26T13:55:02.276 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local cluster=cluster1:0 2023-08-26T13:55:02.277 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ shift 2023-08-26T13:55:02.277 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local instance 2023-08-26T13:55:02.277 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ set_cluster_instance cluster1:0 cluster instance 2023-08-26T13:55:02.277 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local val=cluster1:0 2023-08-26T13:55:02.277 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local cluster_var_name=cluster 2023-08-26T13:55:02.278 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local instance_var_name=instance 2023-08-26T13:55:02.278 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ cluster=cluster1 2023-08-26T13:55:02.278 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ instance=0 2023-08-26T13:55:02.278 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ '[' 0 = cluster1:0 ']' 2023-08-26T13:55:02.278 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ eval cluster=cluster1 2023-08-26T13:55:02.279 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ cluster=cluster1 2023-08-26T13:55:02.280 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ eval instance=0 2023-08-26T13:55:02.281 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ instance=0 2023-08-26T13:55:02.281 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ daemon_asok_file cluster1:0 cluster1 2023-08-26T13:55:02.281 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ local local_cluster=cluster1:0 2023-08-26T13:55:02.281 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ local cluster=cluster1 2023-08-26T13:55:02.281 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ local instance 2023-08-26T13:55:02.281 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2023-08-26T13:55:02.282 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ local val=cluster1:0 2023-08-26T13:55:02.282 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ local cluster_var_name=local_cluster 2023-08-26T13:55:02.282 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ local instance_var_name=instance 2023-08-26T13:55:02.282 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ cluster=cluster1 2023-08-26T13:55:02.282 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ instance=0 2023-08-26T13:55:02.283 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ '[' 0 = cluster1:0 ']' 2023-08-26T13:55:02.283 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ eval local_cluster=cluster1 2023-08-26T13:55:02.283 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++++ local_cluster=cluster1 2023-08-26T13:55:02.283 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ eval instance=0 2023-08-26T13:55:02.283 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++++ instance=0 2023-08-26T13:55:02.284 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2023-08-26T13:55:02.295 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-08-26T13:55:02.296 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-08-26T13:55:02.296 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-08-26T13:55:02.296 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_20 2023-08-26T13:55:02.408 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:Invalid command: unused arguments: ['mirror/image_20'] 2023-08-26T13:55:02.408 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:rbd mirror status : get status for rbd mirror 2023-08-26T13:55:02.408 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:admin_socket: invalid command 2023-08-26T13:55:02.411 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-08-26T13:55:02.411 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_20 2023-08-26T13:55:02.411 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local cluster=cluster1:1 2023-08-26T13:55:02.412 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ shift 2023-08-26T13:55:02.412 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local instance 2023-08-26T13:55:02.412 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ set_cluster_instance cluster1:1 cluster instance 2023-08-26T13:55:02.412 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local val=cluster1:1 2023-08-26T13:55:02.412 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local cluster_var_name=cluster 2023-08-26T13:55:02.412 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local instance_var_name=instance 2023-08-26T13:55:02.413 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ cluster=cluster1 2023-08-26T13:55:02.413 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ instance=1 2023-08-26T13:55:02.413 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ '[' 1 = cluster1:1 ']' 2023-08-26T13:55:02.413 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ eval cluster=cluster1 2023-08-26T13:55:02.413 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ cluster=cluster1 2023-08-26T13:55:02.414 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ eval instance=1 2023-08-26T13:55:02.414 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ instance=1 2023-08-26T13:55:02.414 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ daemon_asok_file cluster1:1 cluster1 2023-08-26T13:55:02.414 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ local local_cluster=cluster1:1 2023-08-26T13:55:02.414 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ local cluster=cluster1 2023-08-26T13:55:02.415 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ local instance 2023-08-26T13:55:02.415 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2023-08-26T13:55:02.415 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ local val=cluster1:1 2023-08-26T13:55:02.415 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ local cluster_var_name=local_cluster 2023-08-26T13:55:02.415 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ local instance_var_name=instance 2023-08-26T13:55:02.415 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ cluster=cluster1 2023-08-26T13:55:02.416 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ instance=1 2023-08-26T13:55:02.416 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ '[' 1 = cluster1:1 ']' 2023-08-26T13:55:02.416 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ eval local_cluster=cluster1 2023-08-26T13:55:02.416 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++++ local_cluster=cluster1 2023-08-26T13:55:02.416 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ eval instance=1 2023-08-26T13:55:02.417 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++++ instance=1 2023-08-26T13:55:02.417 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2023-08-26T13:55:02.429 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-08-26T13:55:02.430 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-08-26T13:55:02.430 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-08-26T13:55:02.430 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_20 2023-08-26T13:55:02.535 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:Invalid command: unused arguments: ['mirror/image_20'] 2023-08-26T13:55:02.535 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:rbd mirror status : get status for rbd mirror 2023-08-26T13:55:02.536 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:admin_socket: invalid command 2023-08-26T13:55:02.538 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-08-26T13:55:02.538 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/image_20 2023-08-26T13:55:02.539 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local cluster=cluster1:2 2023-08-26T13:55:02.539 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ shift 2023-08-26T13:55:02.539 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local instance 2023-08-26T13:55:02.539 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ set_cluster_instance cluster1:2 cluster instance 2023-08-26T13:55:02.539 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local val=cluster1:2 2023-08-26T13:55:02.540 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local cluster_var_name=cluster 2023-08-26T13:55:02.540 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local instance_var_name=instance 2023-08-26T13:55:02.540 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ cluster=cluster1 2023-08-26T13:55:02.540 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ instance=2 2023-08-26T13:55:02.540 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ '[' 2 = cluster1:2 ']' 2023-08-26T13:55:02.540 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ eval cluster=cluster1 2023-08-26T13:55:02.541 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ cluster=cluster1 2023-08-26T13:55:02.541 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ eval instance=2 2023-08-26T13:55:02.541 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ instance=2 2023-08-26T13:55:02.541 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ daemon_asok_file cluster1:2 cluster1 2023-08-26T13:55:02.541 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ local local_cluster=cluster1:2 2023-08-26T13:55:02.541 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ local cluster=cluster1 2023-08-26T13:55:02.542 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ local instance 2023-08-26T13:55:02.542 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2023-08-26T13:55:02.542 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ local val=cluster1:2 2023-08-26T13:55:02.542 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ local cluster_var_name=local_cluster 2023-08-26T13:55:02.542 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ local instance_var_name=instance 2023-08-26T13:55:02.543 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ cluster=cluster1 2023-08-26T13:55:02.543 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ instance=2 2023-08-26T13:55:02.543 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ '[' 2 = cluster1:2 ']' 2023-08-26T13:55:02.543 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ eval local_cluster=cluster1 2023-08-26T13:55:02.543 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++++ local_cluster=cluster1 2023-08-26T13:55:02.544 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ eval instance=2 2023-08-26T13:55:02.544 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++++ instance=2 2023-08-26T13:55:02.544 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2023-08-26T13:55:02.557 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-08-26T13:55:02.557 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-08-26T13:55:02.557 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-08-26T13:55:02.557 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/image_20 2023-08-26T13:55:02.663 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:Invalid command: unused arguments: ['mirror/image_20'] 2023-08-26T13:55:02.663 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:rbd mirror status : get status for rbd mirror 2023-08-26T13:55:02.663 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:admin_socket: invalid command 2023-08-26T13:55:02.666 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-08-26T13:55:02.666 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ admin_daemon cluster1:3 rbd mirror status mirror/image_20 2023-08-26T13:55:02.666 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local cluster=cluster1:3 2023-08-26T13:55:02.666 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ shift 2023-08-26T13:55:02.667 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local instance 2023-08-26T13:55:02.667 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ set_cluster_instance cluster1:3 cluster instance 2023-08-26T13:55:02.667 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local val=cluster1:3 2023-08-26T13:55:02.667 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local cluster_var_name=cluster 2023-08-26T13:55:02.667 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local instance_var_name=instance 2023-08-26T13:55:02.668 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ cluster=cluster1 2023-08-26T13:55:02.668 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ instance=3 2023-08-26T13:55:02.668 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ '[' 3 = cluster1:3 ']' 2023-08-26T13:55:02.668 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ eval cluster=cluster1 2023-08-26T13:55:02.668 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ cluster=cluster1 2023-08-26T13:55:02.669 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ eval instance=3 2023-08-26T13:55:02.669 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ instance=3 2023-08-26T13:55:02.669 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ daemon_asok_file cluster1:3 cluster1 2023-08-26T13:55:02.669 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ local local_cluster=cluster1:3 2023-08-26T13:55:02.669 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ local cluster=cluster1 2023-08-26T13:55:02.669 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ local instance 2023-08-26T13:55:02.670 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ set_cluster_instance cluster1:3 local_cluster instance 2023-08-26T13:55:02.670 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ local val=cluster1:3 2023-08-26T13:55:02.670 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ local cluster_var_name=local_cluster 2023-08-26T13:55:02.670 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ local instance_var_name=instance 2023-08-26T13:55:02.670 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ cluster=cluster1 2023-08-26T13:55:02.671 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ instance=3 2023-08-26T13:55:02.671 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ '[' 3 = cluster1:3 ']' 2023-08-26T13:55:02.671 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ eval local_cluster=cluster1 2023-08-26T13:55:02.671 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++++ local_cluster=cluster1 2023-08-26T13:55:02.671 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ eval instance=3 2023-08-26T13:55:02.672 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++++ instance=3 2023-08-26T13:55:02.672 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2023-08-26T13:55:02.684 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2023-08-26T13:55:02.685 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2023-08-26T13:55:02.685 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2023-08-26T13:55:02.685 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror status mirror/image_20 2023-08-26T13:55:02.799 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ return 0 2023-08-26T13:55:02.800 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ status_result=' "state": "Replaying"' 2023-08-26T13:55:02.800 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ echo ' "state": "Replaying"' 2023-08-26T13:55:02.800 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ grep -i Replaying 2023-08-26T13:55:02.801 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: "state": "Replaying" 2023-08-26T13:55:02.802 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ current_state=started 2023-08-26T13:55:02.802 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ test started = started 2023-08-26T13:55:02.803 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ return 0 2023-08-26T13:55:02.803 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_20 2023-08-26T13:55:02.803 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local local_cluster=cluster1 2023-08-26T13:55:02.803 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster2 2023-08-26T13:55:02.803 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T13:55:02.804 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=image_20 2023-08-26T13:55:02.804 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ '[' snapshot = journal ']' 2023-08-26T13:55:02.804 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ '[' snapshot = snapshot ']' 2023-08-26T13:55:02.804 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_20 2023-08-26T13:55:02.804 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local local_cluster=cluster1 2023-08-26T13:55:02.805 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster2 2023-08-26T13:55:02.805 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T13:55:02.805 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=image_20 2023-08-26T13:55:02.805 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ mkfname cluster2-mirror-image_20.status 2023-08-26T13:55:02.805 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ echo cluster2-mirror-image_20.status 2023-08-26T13:55:02.805 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ sed -e 's|[/ ]|_|g' 2023-08-26T13:55:02.807 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local status_log=/tmp/tmp.TfDn6BcQ3N/cluster2-mirror-image_20.status 2023-08-26T13:55:02.807 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ mkfname cluster1-mirror-image_20.status 2023-08-26T13:55:02.807 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ echo cluster1-mirror-image_20.status 2023-08-26T13:55:02.807 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ sed -e 's|[/ ]|_|g' 2023-08-26T13:55:02.807 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local local_status_log=/tmp/tmp.TfDn6BcQ3N/cluster1-mirror-image_20.status 2023-08-26T13:55:02.808 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ mirror_image_snapshot cluster2 mirror image_20 2023-08-26T13:55:02.808 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster2 2023-08-26T13:55:02.808 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T13:55:02.808 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=image_20 2023-08-26T13:55:02.808 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_20 2023-08-26T13:55:03.275 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:Snapshot ID: 157 2023-08-26T13:55:03.283 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_20 /tmp/tmp.TfDn6BcQ3N/cluster2-mirror-image_20.status 2023-08-26T13:55:03.283 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster2 2023-08-26T13:55:03.283 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T13:55:03.283 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=image_20 2023-08-26T13:55:03.284 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local log=/tmp/tmp.TfDn6BcQ3N/cluster2-mirror-image_20.status 2023-08-26T13:55:03.284 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_20 --format xml 2023-08-26T13:55:03.284 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-08-26T13:55:03.326 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ xmlstarlet sel -t -v //snapshot/id 2023-08-26T13:55:03.328 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local snapshot_id=157 2023-08-26T13:55:03.328 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ true 2023-08-26T13:55:03.328 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-08-26T13:55:03.328 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ sleep 0.2 2023-08-26T13:55:03.529 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_20 /tmp/tmp.TfDn6BcQ3N/cluster1-mirror-image_20.status 2023-08-26T13:55:03.529 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster1 2023-08-26T13:55:03.530 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T13:55:03.530 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=image_20 2023-08-26T13:55:03.530 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local log=/tmp/tmp.TfDn6BcQ3N/cluster1-mirror-image_20.status 2023-08-26T13:55:03.530 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_20 --format xml 2023-08-26T13:55:03.530 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-08-26T13:55:03.576 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ true 2023-08-26T13:55:03.576 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-08-26T13:55:03.577 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:-:1.1: Document is empty 2023-08-26T13:55:03.577 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr: 2023-08-26T13:55:03.577 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:^ 2023-08-26T13:55:03.578 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local primary_snapshot_id= 2023-08-26T13:55:03.578 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ test 157 = '' 2023-08-26T13:55:03.579 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-08-26T13:55:03.579 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ sleep 0.4 2023-08-26T13:55:03.979 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_20 /tmp/tmp.TfDn6BcQ3N/cluster1-mirror-image_20.status 2023-08-26T13:55:03.979 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster1 2023-08-26T13:55:03.979 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T13:55:03.980 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=image_20 2023-08-26T13:55:03.980 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local log=/tmp/tmp.TfDn6BcQ3N/cluster1-mirror-image_20.status 2023-08-26T13:55:03.980 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_20 --format xml 2023-08-26T13:55:03.980 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-08-26T13:55:04.025 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ true 2023-08-26T13:55:04.026 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-08-26T13:55:04.027 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:-:1.1: Document is empty 2023-08-26T13:55:04.027 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr: 2023-08-26T13:55:04.027 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:^ 2023-08-26T13:55:04.028 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local primary_snapshot_id= 2023-08-26T13:55:04.028 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ test 157 = '' 2023-08-26T13:55:04.028 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-08-26T13:55:04.028 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ sleep 0.8 2023-08-26T13:55:04.829 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_20 /tmp/tmp.TfDn6BcQ3N/cluster1-mirror-image_20.status 2023-08-26T13:55:04.829 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster1 2023-08-26T13:55:04.829 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T13:55:04.830 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=image_20 2023-08-26T13:55:04.830 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local log=/tmp/tmp.TfDn6BcQ3N/cluster1-mirror-image_20.status 2023-08-26T13:55:04.830 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_20 --format xml 2023-08-26T13:55:04.830 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-08-26T13:55:04.874 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-08-26T13:55:04.876 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local primary_snapshot_id=157 2023-08-26T13:55:04.876 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ test 157 = 157 2023-08-26T13:55:04.877 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ return 0 2023-08-26T13:55:04.877 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ wait_for_snap_present cluster1 mirror image_20 snap 2023-08-26T13:55:04.877 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster1 2023-08-26T13:55:04.877 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T13:55:04.877 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=image_20 2023-08-26T13:55:04.878 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local snap_name=snap 2023-08-26T13:55:04.878 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local s 2023-08-26T13:55:04.878 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2023-08-26T13:55:04.878 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ sleep 1 2023-08-26T13:55:05.879 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rbd --cluster cluster1 info mirror/image_20@snap 2023-08-26T13:55:05.920 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:rbd image 'image_20': 2023-08-26T13:55:05.920 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: size 128 MiB in 32 objects 2023-08-26T13:55:05.921 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: order 22 (4 MiB objects) 2023-08-26T13:55:05.921 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: snapshot_count: 2 2023-08-26T13:55:05.921 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: id: 12b63f4a457a 2023-08-26T13:55:05.921 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: block_name_prefix: rbd_data.12b63f4a457a 2023-08-26T13:55:05.922 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: format: 2 2023-08-26T13:55:05.922 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: features: layering, exclusive-lock, non-primary 2023-08-26T13:55:05.922 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: op_features: 2023-08-26T13:55:05.922 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: flags: 2023-08-26T13:55:05.922 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: create_timestamp: Sat Aug 26 13:45:55 2023 2023-08-26T13:55:05.922 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: access_timestamp: Sat Aug 26 13:45:55 2023 2023-08-26T13:55:05.923 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: modify_timestamp: Sat Aug 26 13:45:55 2023 2023-08-26T13:55:05.923 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: protected: False 2023-08-26T13:55:05.923 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: mirroring state: enabled 2023-08-26T13:55:05.923 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: mirroring mode: snapshot 2023-08-26T13:55:05.924 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: mirroring global id: e8a98866-d970-4e86-bb3b-2de28303ceff 2023-08-26T13:55:05.924 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: mirroring primary: false 2023-08-26T13:55:05.925 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ return 0 2023-08-26T13:55:05.925 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ compare_image_snaps mirror image_20 snap 2023-08-26T13:55:05.925 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T13:55:05.925 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=image_20 2023-08-26T13:55:05.926 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local snap_name=snap 2023-08-26T13:55:05.926 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local ret=0 2023-08-26T13:55:05.926 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local rmt_export=/tmp/tmp.TfDn6BcQ3N/cluster2-mirror-image_20.export 2023-08-26T13:55:05.926 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local loc_export=/tmp/tmp.TfDn6BcQ3N/cluster1-mirror-image_20.export 2023-08-26T13:55:05.926 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rm -f /tmp/tmp.TfDn6BcQ3N/cluster2-mirror-image_20.export /tmp/tmp.TfDn6BcQ3N/cluster1-mirror-image_20.export 2023-08-26T13:55:05.927 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rbd --cluster cluster2 -p mirror export image_20@snap /tmp/tmp.TfDn6BcQ3N/cluster2-mirror-image_20.export 2023-08-26T13:55:06.106 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-08-26T13:55:06.112 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rbd --cluster cluster1 -p mirror export image_20@snap /tmp/tmp.TfDn6BcQ3N/cluster1-mirror-image_20.export 2023-08-26T13:55:06.349 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-08-26T13:55:06.355 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ cmp /tmp/tmp.TfDn6BcQ3N/cluster2-mirror-image_20.export /tmp/tmp.TfDn6BcQ3N/cluster1-mirror-image_20.export 2023-08-26T13:55:06.410 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rm -f /tmp/tmp.TfDn6BcQ3N/cluster2-mirror-image_20.export /tmp/tmp.TfDn6BcQ3N/cluster1-mirror-image_20.export 2023-08-26T13:55:06.446 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ return 0 2023-08-26T13:55:06.447 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-08-26T13:55:06.447 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ image=image_21 2023-08-26T13:55:06.447 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ create_snap cluster2 mirror image_21 snap 2023-08-26T13:55:06.447 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster2 2023-08-26T13:55:06.448 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T13:55:06.448 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=image_21 2023-08-26T13:55:06.448 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local snap_name=snap 2023-08-26T13:55:06.448 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rbd --cluster cluster2 -p mirror snap create image_21@snap --debug-rbd=20 --debug-journaler=20 2023-08-26T13:55:08.284 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ wait_for_image_replay_started cluster1 mirror image_21 2023-08-26T13:55:08.284 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster1 2023-08-26T13:55:08.284 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T13:55:08.284 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=image_21 2023-08-26T13:55:08.285 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ wait_for_image_replay_state cluster1 mirror image_21 started 2023-08-26T13:55:08.285 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster1 2023-08-26T13:55:08.285 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T13:55:08.285 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=image_21 2023-08-26T13:55:08.285 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local state=started 2023-08-26T13:55:08.286 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local s 2023-08-26T13:55:08.286 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-08-26T13:55:08.286 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ sleep 1 2023-08-26T13:55:09.286 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ test_image_replay_state cluster1 mirror image_21 started 2023-08-26T13:55:09.286 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster1 2023-08-26T13:55:09.286 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T13:55:09.287 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=image_21 2023-08-26T13:55:09.287 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local test_state=started 2023-08-26T13:55:09.287 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local status_result 2023-08-26T13:55:09.287 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local current_state=stopped 2023-08-26T13:55:09.287 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_21 2023-08-26T13:55:09.288 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ grep -i state 2023-08-26T13:55:09.288 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local cluster_instance=cluster1 2023-08-26T13:55:09.288 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ shift 2023-08-26T13:55:09.288 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local cluster=cluster1 2023-08-26T13:55:09.288 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local instance=cluster1 2023-08-26T13:55:09.288 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local loop_instance 2023-08-26T13:55:09.289 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-08-26T13:55:09.289 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ sleep 0 2023-08-26T13:55:09.290 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ '[' cluster1 '!=' cluster1 ']' 2023-08-26T13:55:09.291 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ seq 0 3 2023-08-26T13:55:09.291 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-08-26T13:55:09.291 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_21 2023-08-26T13:55:09.291 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local cluster=cluster1:0 2023-08-26T13:55:09.292 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ shift 2023-08-26T13:55:09.292 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local instance 2023-08-26T13:55:09.292 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ set_cluster_instance cluster1:0 cluster instance 2023-08-26T13:55:09.292 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local val=cluster1:0 2023-08-26T13:55:09.292 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local cluster_var_name=cluster 2023-08-26T13:55:09.292 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local instance_var_name=instance 2023-08-26T13:55:09.293 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ cluster=cluster1 2023-08-26T13:55:09.293 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ instance=0 2023-08-26T13:55:09.293 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ '[' 0 = cluster1:0 ']' 2023-08-26T13:55:09.293 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ eval cluster=cluster1 2023-08-26T13:55:09.293 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ cluster=cluster1 2023-08-26T13:55:09.293 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ eval instance=0 2023-08-26T13:55:09.294 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ instance=0 2023-08-26T13:55:09.294 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ daemon_asok_file cluster1:0 cluster1 2023-08-26T13:55:09.294 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ local local_cluster=cluster1:0 2023-08-26T13:55:09.296 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ local cluster=cluster1 2023-08-26T13:55:09.296 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ local instance 2023-08-26T13:55:09.296 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2023-08-26T13:55:09.296 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ local val=cluster1:0 2023-08-26T13:55:09.296 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ local cluster_var_name=local_cluster 2023-08-26T13:55:09.297 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ local instance_var_name=instance 2023-08-26T13:55:09.297 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ cluster=cluster1 2023-08-26T13:55:09.297 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ instance=0 2023-08-26T13:55:09.297 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ '[' 0 = cluster1:0 ']' 2023-08-26T13:55:09.297 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ eval local_cluster=cluster1 2023-08-26T13:55:09.297 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++++ local_cluster=cluster1 2023-08-26T13:55:09.298 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ eval instance=0 2023-08-26T13:55:09.298 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++++ instance=0 2023-08-26T13:55:09.298 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2023-08-26T13:55:09.313 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-08-26T13:55:09.313 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-08-26T13:55:09.313 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-08-26T13:55:09.314 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_21 2023-08-26T13:55:09.424 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:Invalid command: unused arguments: ['mirror/image_21'] 2023-08-26T13:55:09.425 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:rbd mirror status : get status for rbd mirror 2023-08-26T13:55:09.425 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:admin_socket: invalid command 2023-08-26T13:55:09.427 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-08-26T13:55:09.428 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_21 2023-08-26T13:55:09.428 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local cluster=cluster1:1 2023-08-26T13:55:09.428 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ shift 2023-08-26T13:55:09.428 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local instance 2023-08-26T13:55:09.429 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ set_cluster_instance cluster1:1 cluster instance 2023-08-26T13:55:09.429 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local val=cluster1:1 2023-08-26T13:55:09.429 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local cluster_var_name=cluster 2023-08-26T13:55:09.429 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local instance_var_name=instance 2023-08-26T13:55:09.429 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ cluster=cluster1 2023-08-26T13:55:09.430 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ instance=1 2023-08-26T13:55:09.430 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ '[' 1 = cluster1:1 ']' 2023-08-26T13:55:09.430 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ eval cluster=cluster1 2023-08-26T13:55:09.430 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ cluster=cluster1 2023-08-26T13:55:09.430 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ eval instance=1 2023-08-26T13:55:09.431 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ instance=1 2023-08-26T13:55:09.431 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ daemon_asok_file cluster1:1 cluster1 2023-08-26T13:55:09.431 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ local local_cluster=cluster1:1 2023-08-26T13:55:09.431 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ local cluster=cluster1 2023-08-26T13:55:09.431 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ local instance 2023-08-26T13:55:09.432 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2023-08-26T13:55:09.432 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ local val=cluster1:1 2023-08-26T13:55:09.432 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ local cluster_var_name=local_cluster 2023-08-26T13:55:09.432 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ local instance_var_name=instance 2023-08-26T13:55:09.432 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ cluster=cluster1 2023-08-26T13:55:09.433 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ instance=1 2023-08-26T13:55:09.433 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ '[' 1 = cluster1:1 ']' 2023-08-26T13:55:09.433 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ eval local_cluster=cluster1 2023-08-26T13:55:09.433 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++++ local_cluster=cluster1 2023-08-26T13:55:09.433 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ eval instance=1 2023-08-26T13:55:09.433 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++++ instance=1 2023-08-26T13:55:09.434 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2023-08-26T13:55:09.448 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-08-26T13:55:09.448 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-08-26T13:55:09.448 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-08-26T13:55:09.449 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_21 2023-08-26T13:55:09.554 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:Invalid command: unused arguments: ['mirror/image_21'] 2023-08-26T13:55:09.554 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:rbd mirror status : get status for rbd mirror 2023-08-26T13:55:09.554 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:admin_socket: invalid command 2023-08-26T13:55:09.557 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-08-26T13:55:09.557 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/image_21 2023-08-26T13:55:09.557 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local cluster=cluster1:2 2023-08-26T13:55:09.558 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ shift 2023-08-26T13:55:09.558 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local instance 2023-08-26T13:55:09.558 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ set_cluster_instance cluster1:2 cluster instance 2023-08-26T13:55:09.558 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local val=cluster1:2 2023-08-26T13:55:09.558 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local cluster_var_name=cluster 2023-08-26T13:55:09.559 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local instance_var_name=instance 2023-08-26T13:55:09.559 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ cluster=cluster1 2023-08-26T13:55:09.559 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ instance=2 2023-08-26T13:55:09.559 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ '[' 2 = cluster1:2 ']' 2023-08-26T13:55:09.559 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ eval cluster=cluster1 2023-08-26T13:55:09.560 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ cluster=cluster1 2023-08-26T13:55:09.560 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ eval instance=2 2023-08-26T13:55:09.560 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ instance=2 2023-08-26T13:55:09.560 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ daemon_asok_file cluster1:2 cluster1 2023-08-26T13:55:09.561 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ local local_cluster=cluster1:2 2023-08-26T13:55:09.561 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ local cluster=cluster1 2023-08-26T13:55:09.561 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ local instance 2023-08-26T13:55:09.561 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2023-08-26T13:55:09.561 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ local val=cluster1:2 2023-08-26T13:55:09.562 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ local cluster_var_name=local_cluster 2023-08-26T13:55:09.562 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ local instance_var_name=instance 2023-08-26T13:55:09.562 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ cluster=cluster1 2023-08-26T13:55:09.562 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ instance=2 2023-08-26T13:55:09.562 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ '[' 2 = cluster1:2 ']' 2023-08-26T13:55:09.563 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ eval local_cluster=cluster1 2023-08-26T13:55:09.563 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++++ local_cluster=cluster1 2023-08-26T13:55:09.563 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ eval instance=2 2023-08-26T13:55:09.563 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++++ instance=2 2023-08-26T13:55:09.563 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2023-08-26T13:55:09.576 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-08-26T13:55:09.576 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-08-26T13:55:09.576 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-08-26T13:55:09.576 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/image_21 2023-08-26T13:55:09.690 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ return 0 2023-08-26T13:55:09.691 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ status_result=' "state": "Replaying"' 2023-08-26T13:55:09.691 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ echo ' "state": "Replaying"' 2023-08-26T13:55:09.691 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ grep -i Replaying 2023-08-26T13:55:09.692 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: "state": "Replaying" 2023-08-26T13:55:09.694 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ current_state=started 2023-08-26T13:55:09.694 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ test started = started 2023-08-26T13:55:09.694 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ return 0 2023-08-26T13:55:09.694 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_21 2023-08-26T13:55:09.695 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local local_cluster=cluster1 2023-08-26T13:55:09.695 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster2 2023-08-26T13:55:09.695 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T13:55:09.695 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=image_21 2023-08-26T13:55:09.695 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ '[' snapshot = journal ']' 2023-08-26T13:55:09.696 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ '[' snapshot = snapshot ']' 2023-08-26T13:55:09.696 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_21 2023-08-26T13:55:09.696 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local local_cluster=cluster1 2023-08-26T13:55:09.696 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster2 2023-08-26T13:55:09.696 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T13:55:09.697 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=image_21 2023-08-26T13:55:09.697 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ mkfname cluster2-mirror-image_21.status 2023-08-26T13:55:09.697 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ echo cluster2-mirror-image_21.status 2023-08-26T13:55:09.697 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ sed -e 's|[/ ]|_|g' 2023-08-26T13:55:09.697 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local status_log=/tmp/tmp.TfDn6BcQ3N/cluster2-mirror-image_21.status 2023-08-26T13:55:09.697 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ mkfname cluster1-mirror-image_21.status 2023-08-26T13:55:09.698 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ echo cluster1-mirror-image_21.status 2023-08-26T13:55:09.698 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ sed -e 's|[/ ]|_|g' 2023-08-26T13:55:09.699 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local local_status_log=/tmp/tmp.TfDn6BcQ3N/cluster1-mirror-image_21.status 2023-08-26T13:55:09.699 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ mirror_image_snapshot cluster2 mirror image_21 2023-08-26T13:55:09.699 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster2 2023-08-26T13:55:09.699 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T13:55:09.700 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=image_21 2023-08-26T13:55:09.700 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_21 2023-08-26T13:55:10.285 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:Snapshot ID: 159 2023-08-26T13:55:10.294 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_21 /tmp/tmp.TfDn6BcQ3N/cluster2-mirror-image_21.status 2023-08-26T13:55:10.294 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster2 2023-08-26T13:55:10.294 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T13:55:10.295 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=image_21 2023-08-26T13:55:10.295 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local log=/tmp/tmp.TfDn6BcQ3N/cluster2-mirror-image_21.status 2023-08-26T13:55:10.295 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_21 --format xml 2023-08-26T13:55:10.295 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-08-26T13:55:10.337 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ xmlstarlet sel -t -v //snapshot/id 2023-08-26T13:55:10.339 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local snapshot_id=159 2023-08-26T13:55:10.339 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ true 2023-08-26T13:55:10.340 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-08-26T13:55:10.340 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ sleep 0.2 2023-08-26T13:55:10.540 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_21 /tmp/tmp.TfDn6BcQ3N/cluster1-mirror-image_21.status 2023-08-26T13:55:10.541 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster1 2023-08-26T13:55:10.541 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T13:55:10.541 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=image_21 2023-08-26T13:55:10.541 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local log=/tmp/tmp.TfDn6BcQ3N/cluster1-mirror-image_21.status 2023-08-26T13:55:10.542 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_21 --format xml 2023-08-26T13:55:10.542 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-08-26T13:55:10.586 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-08-26T13:55:10.588 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local primary_snapshot_id=88 2023-08-26T13:55:10.588 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ test 159 = 88 2023-08-26T13:55:10.588 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-08-26T13:55:10.588 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ sleep 0.4 2023-08-26T13:55:10.989 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_21 /tmp/tmp.TfDn6BcQ3N/cluster1-mirror-image_21.status 2023-08-26T13:55:10.989 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster1 2023-08-26T13:55:10.990 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T13:55:10.990 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=image_21 2023-08-26T13:55:10.990 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local log=/tmp/tmp.TfDn6BcQ3N/cluster1-mirror-image_21.status 2023-08-26T13:55:10.990 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_21 --format xml 2023-08-26T13:55:10.990 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-08-26T13:55:11.036 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-08-26T13:55:11.038 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local primary_snapshot_id=88 2023-08-26T13:55:11.038 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ test 159 = 88 2023-08-26T13:55:11.039 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-08-26T13:55:11.039 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ sleep 0.8 2023-08-26T13:55:11.840 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_21 /tmp/tmp.TfDn6BcQ3N/cluster1-mirror-image_21.status 2023-08-26T13:55:11.840 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster1 2023-08-26T13:55:11.841 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T13:55:11.841 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=image_21 2023-08-26T13:55:11.841 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local log=/tmp/tmp.TfDn6BcQ3N/cluster1-mirror-image_21.status 2023-08-26T13:55:11.841 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_21 --format xml 2023-08-26T13:55:11.841 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-08-26T13:55:11.891 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ true 2023-08-26T13:55:11.891 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-08-26T13:55:11.892 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:-:1.1: Document is empty 2023-08-26T13:55:11.892 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr: 2023-08-26T13:55:11.893 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:^ 2023-08-26T13:55:11.893 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local primary_snapshot_id= 2023-08-26T13:55:11.894 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ test 159 = '' 2023-08-26T13:55:11.894 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-08-26T13:55:11.894 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ sleep 1.6 2023-08-26T13:55:12.745 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.0 with skip (0.8045458060618356) > weight (0.25) 2023-08-26T13:55:12.745 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.1 with skip (0.32499113956611514) > weight (0.25) 2023-08-26T13:55:12.745 INFO:tasks.rbd_mirror_thrash:kill cluster1.client.mirror.2 2023-08-26T13:55:12.746 INFO:tasks.rbd_mirror.cluster1.client.mirror.2:Sent signal 15 2023-08-26T13:55:12.746 INFO:tasks.rbd_mirror_thrash:waiting for 9 secs before reviving daemons 2023-08-26T13:55:12.747 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi124.stderr:2023-08-26T13:55:12.746+0000 7fecaf0c8700 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.2 (PID: 119979) UID: 1000 2023-08-26T13:55:13.494 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_21 /tmp/tmp.TfDn6BcQ3N/cluster1-mirror-image_21.status 2023-08-26T13:55:13.494 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster1 2023-08-26T13:55:13.495 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T13:55:13.495 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=image_21 2023-08-26T13:55:13.495 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local log=/tmp/tmp.TfDn6BcQ3N/cluster1-mirror-image_21.status 2023-08-26T13:55:13.495 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_21 --format xml 2023-08-26T13:55:13.495 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-08-26T13:55:13.540 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-08-26T13:55:13.542 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local primary_snapshot_id=159 2023-08-26T13:55:13.543 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ test 159 = 159 2023-08-26T13:55:13.543 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ return 0 2023-08-26T13:55:13.543 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ wait_for_snap_present cluster1 mirror image_21 snap 2023-08-26T13:55:13.543 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster1 2023-08-26T13:55:13.543 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T13:55:13.544 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=image_21 2023-08-26T13:55:13.544 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local snap_name=snap 2023-08-26T13:55:13.544 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local s 2023-08-26T13:55:13.544 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2023-08-26T13:55:13.544 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ sleep 1 2023-08-26T13:55:14.543 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rbd --cluster cluster1 info mirror/image_21@snap 2023-08-26T13:55:14.586 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:rbd image 'image_21': 2023-08-26T13:55:14.586 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: size 128 MiB in 32 objects 2023-08-26T13:55:14.587 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: order 22 (4 MiB objects) 2023-08-26T13:55:14.587 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: snapshot_count: 2 2023-08-26T13:55:14.587 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: id: 12c052937102 2023-08-26T13:55:14.587 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: block_name_prefix: rbd_data.12c052937102 2023-08-26T13:55:14.587 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: format: 2 2023-08-26T13:55:14.588 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: features: layering, exclusive-lock, non-primary 2023-08-26T13:55:14.588 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: op_features: 2023-08-26T13:55:14.588 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: flags: 2023-08-26T13:55:14.588 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: create_timestamp: Sat Aug 26 13:46:07 2023 2023-08-26T13:55:14.588 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: access_timestamp: Sat Aug 26 13:46:07 2023 2023-08-26T13:55:14.589 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: modify_timestamp: Sat Aug 26 13:46:07 2023 2023-08-26T13:55:14.589 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: protected: False 2023-08-26T13:55:14.589 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: mirroring state: enabled 2023-08-26T13:55:14.589 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: mirroring mode: snapshot 2023-08-26T13:55:14.589 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: mirroring global id: 04903506-ddd2-48b7-835f-3797d11e35d3 2023-08-26T13:55:14.590 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: mirroring primary: false 2023-08-26T13:55:14.591 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ return 0 2023-08-26T13:55:14.591 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ compare_image_snaps mirror image_21 snap 2023-08-26T13:55:14.591 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T13:55:14.591 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=image_21 2023-08-26T13:55:14.592 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local snap_name=snap 2023-08-26T13:55:14.592 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local ret=0 2023-08-26T13:55:14.592 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local rmt_export=/tmp/tmp.TfDn6BcQ3N/cluster2-mirror-image_21.export 2023-08-26T13:55:14.592 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local loc_export=/tmp/tmp.TfDn6BcQ3N/cluster1-mirror-image_21.export 2023-08-26T13:55:14.592 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rm -f /tmp/tmp.TfDn6BcQ3N/cluster2-mirror-image_21.export /tmp/tmp.TfDn6BcQ3N/cluster1-mirror-image_21.export 2023-08-26T13:55:14.593 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rbd --cluster cluster2 -p mirror export image_21@snap /tmp/tmp.TfDn6BcQ3N/cluster2-mirror-image_21.export 2023-08-26T13:55:14.767 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-08-26T13:55:14.773 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rbd --cluster cluster1 -p mirror export image_21@snap /tmp/tmp.TfDn6BcQ3N/cluster1-mirror-image_21.export 2023-08-26T13:55:15.017 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-08-26T13:55:15.023 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ cmp /tmp/tmp.TfDn6BcQ3N/cluster2-mirror-image_21.export /tmp/tmp.TfDn6BcQ3N/cluster1-mirror-image_21.export 2023-08-26T13:55:15.079 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rm -f /tmp/tmp.TfDn6BcQ3N/cluster2-mirror-image_21.export /tmp/tmp.TfDn6BcQ3N/cluster1-mirror-image_21.export 2023-08-26T13:55:15.115 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ return 0 2023-08-26T13:55:15.116 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-08-26T13:55:15.116 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ image=image_22 2023-08-26T13:55:15.116 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ create_snap cluster2 mirror image_22 snap 2023-08-26T13:55:15.116 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster2 2023-08-26T13:55:15.116 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T13:55:15.117 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=image_22 2023-08-26T13:55:15.117 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local snap_name=snap 2023-08-26T13:55:15.117 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rbd --cluster cluster2 -p mirror snap create image_22@snap --debug-rbd=20 --debug-journaler=20 2023-08-26T13:55:17.183 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ wait_for_image_replay_started cluster1 mirror image_22 2023-08-26T13:55:17.184 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster1 2023-08-26T13:55:17.185 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T13:55:17.185 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=image_22 2023-08-26T13:55:17.185 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ wait_for_image_replay_state cluster1 mirror image_22 started 2023-08-26T13:55:17.185 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster1 2023-08-26T13:55:17.186 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T13:55:17.186 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=image_22 2023-08-26T13:55:17.186 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local state=started 2023-08-26T13:55:17.186 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local s 2023-08-26T13:55:17.186 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-08-26T13:55:17.187 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ sleep 1 2023-08-26T13:55:18.185 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ test_image_replay_state cluster1 mirror image_22 started 2023-08-26T13:55:18.185 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster1 2023-08-26T13:55:18.186 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T13:55:18.186 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=image_22 2023-08-26T13:55:18.186 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local test_state=started 2023-08-26T13:55:18.186 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local status_result 2023-08-26T13:55:18.187 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local current_state=stopped 2023-08-26T13:55:18.189 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_22 2023-08-26T13:55:18.189 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ grep -i state 2023-08-26T13:55:18.189 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local cluster_instance=cluster1 2023-08-26T13:55:18.189 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ shift 2023-08-26T13:55:18.190 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local cluster=cluster1 2023-08-26T13:55:18.190 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local instance=cluster1 2023-08-26T13:55:18.190 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local loop_instance 2023-08-26T13:55:18.190 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-08-26T13:55:18.190 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ sleep 0 2023-08-26T13:55:18.191 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ '[' cluster1 '!=' cluster1 ']' 2023-08-26T13:55:18.191 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ seq 0 3 2023-08-26T13:55:18.191 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-08-26T13:55:18.191 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_22 2023-08-26T13:55:18.191 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local cluster=cluster1:0 2023-08-26T13:55:18.192 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ shift 2023-08-26T13:55:18.192 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local instance 2023-08-26T13:55:18.192 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ set_cluster_instance cluster1:0 cluster instance 2023-08-26T13:55:18.192 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local val=cluster1:0 2023-08-26T13:55:18.192 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local cluster_var_name=cluster 2023-08-26T13:55:18.193 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local instance_var_name=instance 2023-08-26T13:55:18.193 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ cluster=cluster1 2023-08-26T13:55:18.193 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ instance=0 2023-08-26T13:55:18.193 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ '[' 0 = cluster1:0 ']' 2023-08-26T13:55:18.193 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ eval cluster=cluster1 2023-08-26T13:55:18.193 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ cluster=cluster1 2023-08-26T13:55:18.194 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ eval instance=0 2023-08-26T13:55:18.194 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ instance=0 2023-08-26T13:55:18.195 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ daemon_asok_file cluster1:0 cluster1 2023-08-26T13:55:18.196 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ local local_cluster=cluster1:0 2023-08-26T13:55:18.196 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ local cluster=cluster1 2023-08-26T13:55:18.196 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ local instance 2023-08-26T13:55:18.196 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2023-08-26T13:55:18.196 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ local val=cluster1:0 2023-08-26T13:55:18.197 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ local cluster_var_name=local_cluster 2023-08-26T13:55:18.197 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ local instance_var_name=instance 2023-08-26T13:55:18.197 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ cluster=cluster1 2023-08-26T13:55:18.197 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ instance=0 2023-08-26T13:55:18.197 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ '[' 0 = cluster1:0 ']' 2023-08-26T13:55:18.197 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ eval local_cluster=cluster1 2023-08-26T13:55:18.198 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++++ local_cluster=cluster1 2023-08-26T13:55:18.198 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ eval instance=0 2023-08-26T13:55:18.198 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++++ instance=0 2023-08-26T13:55:18.198 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2023-08-26T13:55:18.211 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-08-26T13:55:18.211 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-08-26T13:55:18.212 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-08-26T13:55:18.212 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_22 2023-08-26T13:55:18.322 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:Invalid command: unused arguments: ['mirror/image_22'] 2023-08-26T13:55:18.323 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:rbd mirror status : get status for rbd mirror 2023-08-26T13:55:18.323 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:admin_socket: invalid command 2023-08-26T13:55:18.326 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-08-26T13:55:18.326 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_22 2023-08-26T13:55:18.326 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local cluster=cluster1:1 2023-08-26T13:55:18.327 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ shift 2023-08-26T13:55:18.327 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local instance 2023-08-26T13:55:18.327 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ set_cluster_instance cluster1:1 cluster instance 2023-08-26T13:55:18.327 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local val=cluster1:1 2023-08-26T13:55:18.327 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local cluster_var_name=cluster 2023-08-26T13:55:18.327 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local instance_var_name=instance 2023-08-26T13:55:18.328 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ cluster=cluster1 2023-08-26T13:55:18.328 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ instance=1 2023-08-26T13:55:18.328 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ '[' 1 = cluster1:1 ']' 2023-08-26T13:55:18.328 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ eval cluster=cluster1 2023-08-26T13:55:18.328 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ cluster=cluster1 2023-08-26T13:55:18.329 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ eval instance=1 2023-08-26T13:55:18.329 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ instance=1 2023-08-26T13:55:18.329 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ daemon_asok_file cluster1:1 cluster1 2023-08-26T13:55:18.329 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ local local_cluster=cluster1:1 2023-08-26T13:55:18.329 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ local cluster=cluster1 2023-08-26T13:55:18.330 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ local instance 2023-08-26T13:55:18.330 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2023-08-26T13:55:18.330 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ local val=cluster1:1 2023-08-26T13:55:18.330 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ local cluster_var_name=local_cluster 2023-08-26T13:55:18.330 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ local instance_var_name=instance 2023-08-26T13:55:18.330 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ cluster=cluster1 2023-08-26T13:55:18.331 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ instance=1 2023-08-26T13:55:18.331 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ '[' 1 = cluster1:1 ']' 2023-08-26T13:55:18.331 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ eval local_cluster=cluster1 2023-08-26T13:55:18.331 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++++ local_cluster=cluster1 2023-08-26T13:55:18.331 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ eval instance=1 2023-08-26T13:55:18.332 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++++ instance=1 2023-08-26T13:55:18.332 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2023-08-26T13:55:18.344 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-08-26T13:55:18.345 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-08-26T13:55:18.345 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-08-26T13:55:18.345 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_22 2023-08-26T13:55:18.459 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ return 0 2023-08-26T13:55:18.460 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ status_result=' "state": "Replaying"' 2023-08-26T13:55:18.460 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ echo ' "state": "Replaying"' 2023-08-26T13:55:18.460 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ grep -i Replaying 2023-08-26T13:55:18.461 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: "state": "Replaying" 2023-08-26T13:55:18.463 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ current_state=started 2023-08-26T13:55:18.463 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ test started = started 2023-08-26T13:55:18.463 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ return 0 2023-08-26T13:55:18.463 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_22 2023-08-26T13:55:18.463 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local local_cluster=cluster1 2023-08-26T13:55:18.464 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster2 2023-08-26T13:55:18.464 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T13:55:18.464 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=image_22 2023-08-26T13:55:18.464 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ '[' snapshot = journal ']' 2023-08-26T13:55:18.464 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ '[' snapshot = snapshot ']' 2023-08-26T13:55:18.464 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_22 2023-08-26T13:55:18.465 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local local_cluster=cluster1 2023-08-26T13:55:18.465 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster2 2023-08-26T13:55:18.465 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T13:55:18.465 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=image_22 2023-08-26T13:55:18.465 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ mkfname cluster2-mirror-image_22.status 2023-08-26T13:55:18.466 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ echo cluster2-mirror-image_22.status 2023-08-26T13:55:18.466 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ sed -e 's|[/ ]|_|g' 2023-08-26T13:55:18.466 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local status_log=/tmp/tmp.TfDn6BcQ3N/cluster2-mirror-image_22.status 2023-08-26T13:55:18.466 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ mkfname cluster1-mirror-image_22.status 2023-08-26T13:55:18.466 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ echo cluster1-mirror-image_22.status 2023-08-26T13:55:18.467 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ sed -e 's|[/ ]|_|g' 2023-08-26T13:55:18.467 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local local_status_log=/tmp/tmp.TfDn6BcQ3N/cluster1-mirror-image_22.status 2023-08-26T13:55:18.468 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ mirror_image_snapshot cluster2 mirror image_22 2023-08-26T13:55:18.468 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster2 2023-08-26T13:55:18.468 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T13:55:18.468 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=image_22 2023-08-26T13:55:18.468 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_22 2023-08-26T13:55:19.189 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:Snapshot ID: 161 2023-08-26T13:55:19.198 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_22 /tmp/tmp.TfDn6BcQ3N/cluster2-mirror-image_22.status 2023-08-26T13:55:19.199 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster2 2023-08-26T13:55:19.199 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T13:55:19.199 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=image_22 2023-08-26T13:55:19.199 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local log=/tmp/tmp.TfDn6BcQ3N/cluster2-mirror-image_22.status 2023-08-26T13:55:19.200 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_22 --format xml 2023-08-26T13:55:19.200 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-08-26T13:55:19.244 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ xmlstarlet sel -t -v //snapshot/id 2023-08-26T13:55:19.246 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local snapshot_id=161 2023-08-26T13:55:19.246 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ true 2023-08-26T13:55:19.246 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-08-26T13:55:19.246 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ sleep 0.2 2023-08-26T13:55:19.447 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_22 /tmp/tmp.TfDn6BcQ3N/cluster1-mirror-image_22.status 2023-08-26T13:55:19.448 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster1 2023-08-26T13:55:19.448 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T13:55:19.448 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=image_22 2023-08-26T13:55:19.449 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local log=/tmp/tmp.TfDn6BcQ3N/cluster1-mirror-image_22.status 2023-08-26T13:55:19.449 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_22 --format xml 2023-08-26T13:55:19.449 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-08-26T13:55:19.493 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-08-26T13:55:19.495 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local primary_snapshot_id=89 2023-08-26T13:55:19.495 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ test 161 = 89 2023-08-26T13:55:19.496 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-08-26T13:55:19.496 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ sleep 0.4 2023-08-26T13:55:19.897 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_22 /tmp/tmp.TfDn6BcQ3N/cluster1-mirror-image_22.status 2023-08-26T13:55:19.897 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster1 2023-08-26T13:55:19.897 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T13:55:19.897 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=image_22 2023-08-26T13:55:19.898 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local log=/tmp/tmp.TfDn6BcQ3N/cluster1-mirror-image_22.status 2023-08-26T13:55:19.898 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_22 --format xml 2023-08-26T13:55:19.898 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-08-26T13:55:19.946 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ true 2023-08-26T13:55:19.946 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-08-26T13:55:19.947 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:-:1.1: Document is empty 2023-08-26T13:55:19.947 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr: 2023-08-26T13:55:19.947 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:^ 2023-08-26T13:55:19.948 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local primary_snapshot_id= 2023-08-26T13:55:19.948 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ test 161 = '' 2023-08-26T13:55:19.949 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-08-26T13:55:19.949 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ sleep 0.8 2023-08-26T13:55:20.749 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_22 /tmp/tmp.TfDn6BcQ3N/cluster1-mirror-image_22.status 2023-08-26T13:55:20.750 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster1 2023-08-26T13:55:20.750 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T13:55:20.750 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=image_22 2023-08-26T13:55:20.750 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local log=/tmp/tmp.TfDn6BcQ3N/cluster1-mirror-image_22.status 2023-08-26T13:55:20.751 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_22 --format xml 2023-08-26T13:55:20.751 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-08-26T13:55:20.794 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-08-26T13:55:20.796 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local primary_snapshot_id=161 2023-08-26T13:55:20.797 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ test 161 = 161 2023-08-26T13:55:20.797 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ return 0 2023-08-26T13:55:20.797 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ wait_for_snap_present cluster1 mirror image_22 snap 2023-08-26T13:55:20.797 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster1 2023-08-26T13:55:20.797 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T13:55:20.797 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=image_22 2023-08-26T13:55:20.798 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local snap_name=snap 2023-08-26T13:55:20.798 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local s 2023-08-26T13:55:20.798 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2023-08-26T13:55:20.798 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ sleep 1 2023-08-26T13:55:21.747 INFO:tasks.rbd_mirror_thrash:waiting for cluster1.client.mirror.2 2023-08-26T13:55:21.747 INFO:teuthology.orchestra.run:waiting for 600 2023-08-26T13:55:21.748 INFO:tasks.rbd_mirror_thrash:reviving cluster1.client.mirror.2 2023-08-26T13:55:21.748 INFO:tasks.rbd_mirror.cluster1.client.mirror.2:Restarting daemon 2023-08-26T13:55:21.748 DEBUG:teuthology.orchestra.run.smithi124:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.2 2023-08-26T13:55:21.751 INFO:tasks.rbd_mirror.cluster1.client.mirror.2:Started 2023-08-26T13:55:21.751 INFO:tasks.rbd_mirror_thrash:waiting for 64 secs before thrashing 2023-08-26T13:55:21.797 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rbd --cluster cluster1 info mirror/image_22@snap 2023-08-26T13:55:21.840 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:rbd image 'image_22': 2023-08-26T13:55:21.841 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: size 128 MiB in 32 objects 2023-08-26T13:55:21.841 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: order 22 (4 MiB objects) 2023-08-26T13:55:21.841 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: snapshot_count: 2 2023-08-26T13:55:21.841 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: id: 12c076f83f94 2023-08-26T13:55:21.841 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: block_name_prefix: rbd_data.12c076f83f94 2023-08-26T13:55:21.842 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: format: 2 2023-08-26T13:55:21.842 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: features: layering, exclusive-lock, non-primary 2023-08-26T13:55:21.842 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: op_features: 2023-08-26T13:55:21.842 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: flags: 2023-08-26T13:55:21.842 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: create_timestamp: Sat Aug 26 13:46:15 2023 2023-08-26T13:55:21.842 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: access_timestamp: Sat Aug 26 13:46:15 2023 2023-08-26T13:55:21.843 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: modify_timestamp: Sat Aug 26 13:46:15 2023 2023-08-26T13:55:21.843 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: protected: False 2023-08-26T13:55:21.843 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: mirroring state: enabled 2023-08-26T13:55:21.843 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: mirroring mode: snapshot 2023-08-26T13:55:21.843 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: mirroring global id: 70414d4b-2367-4c05-8614-971399383919 2023-08-26T13:55:21.844 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: mirroring primary: false 2023-08-26T13:55:21.845 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ return 0 2023-08-26T13:55:21.845 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ compare_image_snaps mirror image_22 snap 2023-08-26T13:55:21.845 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T13:55:21.845 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=image_22 2023-08-26T13:55:21.846 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local snap_name=snap 2023-08-26T13:55:21.846 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local ret=0 2023-08-26T13:55:21.846 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local rmt_export=/tmp/tmp.TfDn6BcQ3N/cluster2-mirror-image_22.export 2023-08-26T13:55:21.846 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local loc_export=/tmp/tmp.TfDn6BcQ3N/cluster1-mirror-image_22.export 2023-08-26T13:55:21.846 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rm -f /tmp/tmp.TfDn6BcQ3N/cluster2-mirror-image_22.export /tmp/tmp.TfDn6BcQ3N/cluster1-mirror-image_22.export 2023-08-26T13:55:21.847 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rbd --cluster cluster2 -p mirror export image_22@snap /tmp/tmp.TfDn6BcQ3N/cluster2-mirror-image_22.export 2023-08-26T13:55:22.030 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-08-26T13:55:22.036 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rbd --cluster cluster1 -p mirror export image_22@snap /tmp/tmp.TfDn6BcQ3N/cluster1-mirror-image_22.export 2023-08-26T13:55:22.249 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-08-26T13:55:22.255 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ cmp /tmp/tmp.TfDn6BcQ3N/cluster2-mirror-image_22.export /tmp/tmp.TfDn6BcQ3N/cluster1-mirror-image_22.export 2023-08-26T13:55:22.311 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rm -f /tmp/tmp.TfDn6BcQ3N/cluster2-mirror-image_22.export /tmp/tmp.TfDn6BcQ3N/cluster1-mirror-image_22.export 2023-08-26T13:55:22.348 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ return 0 2023-08-26T13:55:22.348 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-08-26T13:55:22.348 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ image=image_23 2023-08-26T13:55:22.348 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ create_snap cluster2 mirror image_23 snap 2023-08-26T13:55:22.349 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster2 2023-08-26T13:55:22.349 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T13:55:22.349 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=image_23 2023-08-26T13:55:22.349 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local snap_name=snap 2023-08-26T13:55:22.349 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rbd --cluster cluster2 -p mirror snap create image_23@snap --debug-rbd=20 --debug-journaler=20 2023-08-26T13:55:24.246 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ wait_for_image_replay_started cluster1 mirror image_23 2023-08-26T13:55:24.247 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster1 2023-08-26T13:55:24.247 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T13:55:24.247 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=image_23 2023-08-26T13:55:24.247 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ wait_for_image_replay_state cluster1 mirror image_23 started 2023-08-26T13:55:24.247 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster1 2023-08-26T13:55:24.248 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T13:55:24.248 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=image_23 2023-08-26T13:55:24.248 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local state=started 2023-08-26T13:55:24.248 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local s 2023-08-26T13:55:24.248 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-08-26T13:55:24.249 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ sleep 1 2023-08-26T13:55:25.247 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ test_image_replay_state cluster1 mirror image_23 started 2023-08-26T13:55:25.247 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster1 2023-08-26T13:55:25.248 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T13:55:25.248 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=image_23 2023-08-26T13:55:25.248 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local test_state=started 2023-08-26T13:55:25.248 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local status_result 2023-08-26T13:55:25.248 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local current_state=stopped 2023-08-26T13:55:25.252 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_23 2023-08-26T13:55:25.252 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ grep -i state 2023-08-26T13:55:25.253 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local cluster_instance=cluster1 2023-08-26T13:55:25.253 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ shift 2023-08-26T13:55:25.253 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local cluster=cluster1 2023-08-26T13:55:25.253 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local instance=cluster1 2023-08-26T13:55:25.253 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local loop_instance 2023-08-26T13:55:25.253 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-08-26T13:55:25.254 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ sleep 0 2023-08-26T13:55:25.254 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ '[' cluster1 '!=' cluster1 ']' 2023-08-26T13:55:25.254 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ seq 0 3 2023-08-26T13:55:25.254 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-08-26T13:55:25.254 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_23 2023-08-26T13:55:25.254 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local cluster=cluster1:0 2023-08-26T13:55:25.255 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ shift 2023-08-26T13:55:25.255 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local instance 2023-08-26T13:55:25.255 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ set_cluster_instance cluster1:0 cluster instance 2023-08-26T13:55:25.255 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local val=cluster1:0 2023-08-26T13:55:25.255 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local cluster_var_name=cluster 2023-08-26T13:55:25.255 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local instance_var_name=instance 2023-08-26T13:55:25.256 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ cluster=cluster1 2023-08-26T13:55:25.256 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ instance=0 2023-08-26T13:55:25.256 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ '[' 0 = cluster1:0 ']' 2023-08-26T13:55:25.256 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ eval cluster=cluster1 2023-08-26T13:55:25.256 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ cluster=cluster1 2023-08-26T13:55:25.257 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ eval instance=0 2023-08-26T13:55:25.257 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ instance=0 2023-08-26T13:55:25.257 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ daemon_asok_file cluster1:0 cluster1 2023-08-26T13:55:25.257 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ local local_cluster=cluster1:0 2023-08-26T13:55:25.257 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ local cluster=cluster1 2023-08-26T13:55:25.257 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ local instance 2023-08-26T13:55:25.258 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2023-08-26T13:55:25.258 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ local val=cluster1:0 2023-08-26T13:55:25.258 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ local cluster_var_name=local_cluster 2023-08-26T13:55:25.258 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ local instance_var_name=instance 2023-08-26T13:55:25.258 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ cluster=cluster1 2023-08-26T13:55:25.258 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ instance=0 2023-08-26T13:55:25.259 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ '[' 0 = cluster1:0 ']' 2023-08-26T13:55:25.259 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ eval local_cluster=cluster1 2023-08-26T13:55:25.259 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++++ local_cluster=cluster1 2023-08-26T13:55:25.259 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ eval instance=0 2023-08-26T13:55:25.260 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++++ instance=0 2023-08-26T13:55:25.260 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2023-08-26T13:55:25.273 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-08-26T13:55:25.273 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-08-26T13:55:25.274 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-08-26T13:55:25.274 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_23 2023-08-26T13:55:25.392 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ return 0 2023-08-26T13:55:25.393 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ status_result=' "state": "Replaying"' 2023-08-26T13:55:25.393 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ echo ' "state": "Replaying"' 2023-08-26T13:55:25.393 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ grep -i Replaying 2023-08-26T13:55:25.394 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: "state": "Replaying" 2023-08-26T13:55:25.396 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ current_state=started 2023-08-26T13:55:25.396 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ test started = started 2023-08-26T13:55:25.397 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ return 0 2023-08-26T13:55:25.397 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_23 2023-08-26T13:55:25.397 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local local_cluster=cluster1 2023-08-26T13:55:25.397 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster2 2023-08-26T13:55:25.397 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T13:55:25.398 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=image_23 2023-08-26T13:55:25.398 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ '[' snapshot = journal ']' 2023-08-26T13:55:25.398 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ '[' snapshot = snapshot ']' 2023-08-26T13:55:25.398 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_23 2023-08-26T13:55:25.399 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local local_cluster=cluster1 2023-08-26T13:55:25.399 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster2 2023-08-26T13:55:25.399 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T13:55:25.399 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=image_23 2023-08-26T13:55:25.399 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ mkfname cluster2-mirror-image_23.status 2023-08-26T13:55:25.400 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ echo cluster2-mirror-image_23.status 2023-08-26T13:55:25.400 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ sed -e 's|[/ ]|_|g' 2023-08-26T13:55:25.400 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local status_log=/tmp/tmp.TfDn6BcQ3N/cluster2-mirror-image_23.status 2023-08-26T13:55:25.400 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ mkfname cluster1-mirror-image_23.status 2023-08-26T13:55:25.400 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ echo cluster1-mirror-image_23.status 2023-08-26T13:55:25.400 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ sed -e 's|[/ ]|_|g' 2023-08-26T13:55:25.401 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local local_status_log=/tmp/tmp.TfDn6BcQ3N/cluster1-mirror-image_23.status 2023-08-26T13:55:25.402 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ mirror_image_snapshot cluster2 mirror image_23 2023-08-26T13:55:25.402 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster2 2023-08-26T13:55:25.402 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T13:55:25.402 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=image_23 2023-08-26T13:55:25.402 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_23 2023-08-26T13:55:26.256 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:Snapshot ID: 163 2023-08-26T13:55:26.265 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_23 /tmp/tmp.TfDn6BcQ3N/cluster2-mirror-image_23.status 2023-08-26T13:55:26.265 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster2 2023-08-26T13:55:26.265 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T13:55:26.266 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=image_23 2023-08-26T13:55:26.266 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local log=/tmp/tmp.TfDn6BcQ3N/cluster2-mirror-image_23.status 2023-08-26T13:55:26.266 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_23 --format xml 2023-08-26T13:55:26.266 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-08-26T13:55:26.309 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ xmlstarlet sel -t -v //snapshot/id 2023-08-26T13:55:26.311 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local snapshot_id=163 2023-08-26T13:55:26.311 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ true 2023-08-26T13:55:26.311 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-08-26T13:55:26.311 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ sleep 0.2 2023-08-26T13:55:26.512 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_23 /tmp/tmp.TfDn6BcQ3N/cluster1-mirror-image_23.status 2023-08-26T13:55:26.513 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster1 2023-08-26T13:55:26.513 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T13:55:26.513 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=image_23 2023-08-26T13:55:26.513 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local log=/tmp/tmp.TfDn6BcQ3N/cluster1-mirror-image_23.status 2023-08-26T13:55:26.513 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_23 --format xml 2023-08-26T13:55:26.513 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-08-26T13:55:26.558 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ true 2023-08-26T13:55:26.559 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-08-26T13:55:26.560 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:-:1.1: Document is empty 2023-08-26T13:55:26.560 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr: 2023-08-26T13:55:26.560 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:^ 2023-08-26T13:55:26.561 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local primary_snapshot_id= 2023-08-26T13:55:26.561 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ test 163 = '' 2023-08-26T13:55:26.561 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-08-26T13:55:26.562 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ sleep 0.4 2023-08-26T13:55:26.962 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_23 /tmp/tmp.TfDn6BcQ3N/cluster1-mirror-image_23.status 2023-08-26T13:55:26.962 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster1 2023-08-26T13:55:26.962 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T13:55:26.963 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=image_23 2023-08-26T13:55:26.963 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local log=/tmp/tmp.TfDn6BcQ3N/cluster1-mirror-image_23.status 2023-08-26T13:55:26.963 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_23 --format xml 2023-08-26T13:55:26.963 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-08-26T13:55:27.008 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ true 2023-08-26T13:55:27.008 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-08-26T13:55:27.009 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:-:1.1: Document is empty 2023-08-26T13:55:27.010 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr: 2023-08-26T13:55:27.010 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:^ 2023-08-26T13:55:27.010 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local primary_snapshot_id= 2023-08-26T13:55:27.011 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ test 163 = '' 2023-08-26T13:55:27.011 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-08-26T13:55:27.011 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ sleep 0.8 2023-08-26T13:55:27.811 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_23 /tmp/tmp.TfDn6BcQ3N/cluster1-mirror-image_23.status 2023-08-26T13:55:27.812 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster1 2023-08-26T13:55:27.812 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T13:55:27.812 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=image_23 2023-08-26T13:55:27.812 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local log=/tmp/tmp.TfDn6BcQ3N/cluster1-mirror-image_23.status 2023-08-26T13:55:27.813 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_23 --format xml 2023-08-26T13:55:27.813 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-08-26T13:55:27.859 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-08-26T13:55:27.861 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local primary_snapshot_id=163 2023-08-26T13:55:27.861 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ test 163 = 163 2023-08-26T13:55:27.862 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ return 0 2023-08-26T13:55:27.862 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ wait_for_snap_present cluster1 mirror image_23 snap 2023-08-26T13:55:27.862 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster1 2023-08-26T13:55:27.862 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T13:55:27.862 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=image_23 2023-08-26T13:55:27.863 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local snap_name=snap 2023-08-26T13:55:27.863 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local s 2023-08-26T13:55:27.863 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2023-08-26T13:55:27.863 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ sleep 1 2023-08-26T13:55:28.862 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rbd --cluster cluster1 info mirror/image_23@snap 2023-08-26T13:55:28.907 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:rbd image 'image_23': 2023-08-26T13:55:28.907 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: size 128 MiB in 32 objects 2023-08-26T13:55:28.907 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: order 22 (4 MiB objects) 2023-08-26T13:55:28.907 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: snapshot_count: 2 2023-08-26T13:55:28.908 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: id: 12b6d7d6e733 2023-08-26T13:55:28.908 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: block_name_prefix: rbd_data.12b6d7d6e733 2023-08-26T13:55:28.908 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: format: 2 2023-08-26T13:55:28.908 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: features: layering, exclusive-lock, non-primary 2023-08-26T13:55:28.908 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: op_features: 2023-08-26T13:55:28.909 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: flags: 2023-08-26T13:55:28.909 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: create_timestamp: Sat Aug 26 13:46:23 2023 2023-08-26T13:55:28.909 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: access_timestamp: Sat Aug 26 13:46:23 2023 2023-08-26T13:55:28.909 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: modify_timestamp: Sat Aug 26 13:46:23 2023 2023-08-26T13:55:28.909 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: protected: False 2023-08-26T13:55:28.910 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: mirroring state: enabled 2023-08-26T13:55:28.910 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: mirroring mode: snapshot 2023-08-26T13:55:28.910 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: mirroring global id: ba978cfa-fbe5-4a3a-86fa-f20310f2d99f 2023-08-26T13:55:28.910 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: mirroring primary: false 2023-08-26T13:55:28.911 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ return 0 2023-08-26T13:55:28.911 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ compare_image_snaps mirror image_23 snap 2023-08-26T13:55:28.912 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T13:55:28.912 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=image_23 2023-08-26T13:55:28.912 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local snap_name=snap 2023-08-26T13:55:28.912 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local ret=0 2023-08-26T13:55:28.912 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local rmt_export=/tmp/tmp.TfDn6BcQ3N/cluster2-mirror-image_23.export 2023-08-26T13:55:28.913 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local loc_export=/tmp/tmp.TfDn6BcQ3N/cluster1-mirror-image_23.export 2023-08-26T13:55:28.913 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rm -f /tmp/tmp.TfDn6BcQ3N/cluster2-mirror-image_23.export /tmp/tmp.TfDn6BcQ3N/cluster1-mirror-image_23.export 2023-08-26T13:55:28.913 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rbd --cluster cluster2 -p mirror export image_23@snap /tmp/tmp.TfDn6BcQ3N/cluster2-mirror-image_23.export 2023-08-26T13:55:29.086 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-08-26T13:55:29.092 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rbd --cluster cluster1 -p mirror export image_23@snap /tmp/tmp.TfDn6BcQ3N/cluster1-mirror-image_23.export 2023-08-26T13:55:29.310 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-08-26T13:55:29.316 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ cmp /tmp/tmp.TfDn6BcQ3N/cluster2-mirror-image_23.export /tmp/tmp.TfDn6BcQ3N/cluster1-mirror-image_23.export 2023-08-26T13:55:29.371 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rm -f /tmp/tmp.TfDn6BcQ3N/cluster2-mirror-image_23.export /tmp/tmp.TfDn6BcQ3N/cluster1-mirror-image_23.export 2023-08-26T13:55:29.408 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ return 0 2023-08-26T13:55:29.408 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-08-26T13:55:29.408 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ image=image_24 2023-08-26T13:55:29.408 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ create_snap cluster2 mirror image_24 snap 2023-08-26T13:55:29.409 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster2 2023-08-26T13:55:29.409 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T13:55:29.409 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=image_24 2023-08-26T13:55:29.409 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local snap_name=snap 2023-08-26T13:55:29.409 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rbd --cluster cluster2 -p mirror snap create image_24@snap --debug-rbd=20 --debug-journaler=20 2023-08-26T13:55:31.260 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ wait_for_image_replay_started cluster1 mirror image_24 2023-08-26T13:55:31.260 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster1 2023-08-26T13:55:31.261 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T13:55:31.261 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=image_24 2023-08-26T13:55:31.261 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ wait_for_image_replay_state cluster1 mirror image_24 started 2023-08-26T13:55:31.261 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster1 2023-08-26T13:55:31.261 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T13:55:31.262 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=image_24 2023-08-26T13:55:31.262 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local state=started 2023-08-26T13:55:31.262 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local s 2023-08-26T13:55:31.262 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-08-26T13:55:31.262 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ sleep 1 2023-08-26T13:55:32.261 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ test_image_replay_state cluster1 mirror image_24 started 2023-08-26T13:55:32.261 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster1 2023-08-26T13:55:32.262 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T13:55:32.262 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=image_24 2023-08-26T13:55:32.262 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local test_state=started 2023-08-26T13:55:32.262 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local status_result 2023-08-26T13:55:32.262 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local current_state=stopped 2023-08-26T13:55:32.264 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_24 2023-08-26T13:55:32.264 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local cluster_instance=cluster1 2023-08-26T13:55:32.264 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ grep -i state 2023-08-26T13:55:32.264 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ shift 2023-08-26T13:55:32.264 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local cluster=cluster1 2023-08-26T13:55:32.265 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local instance=cluster1 2023-08-26T13:55:32.265 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local loop_instance 2023-08-26T13:55:32.265 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-08-26T13:55:32.265 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ sleep 0 2023-08-26T13:55:32.265 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ '[' cluster1 '!=' cluster1 ']' 2023-08-26T13:55:32.265 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ seq 0 3 2023-08-26T13:55:32.267 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-08-26T13:55:32.267 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_24 2023-08-26T13:55:32.267 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local cluster=cluster1:0 2023-08-26T13:55:32.268 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ shift 2023-08-26T13:55:32.268 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local instance 2023-08-26T13:55:32.268 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ set_cluster_instance cluster1:0 cluster instance 2023-08-26T13:55:32.268 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local val=cluster1:0 2023-08-26T13:55:32.268 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local cluster_var_name=cluster 2023-08-26T13:55:32.269 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local instance_var_name=instance 2023-08-26T13:55:32.269 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ cluster=cluster1 2023-08-26T13:55:32.269 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ instance=0 2023-08-26T13:55:32.269 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ '[' 0 = cluster1:0 ']' 2023-08-26T13:55:32.269 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ eval cluster=cluster1 2023-08-26T13:55:32.269 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ cluster=cluster1 2023-08-26T13:55:32.270 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ eval instance=0 2023-08-26T13:55:32.270 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ instance=0 2023-08-26T13:55:32.270 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ daemon_asok_file cluster1:0 cluster1 2023-08-26T13:55:32.271 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ local local_cluster=cluster1:0 2023-08-26T13:55:32.272 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ local cluster=cluster1 2023-08-26T13:55:32.272 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ local instance 2023-08-26T13:55:32.272 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2023-08-26T13:55:32.272 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ local val=cluster1:0 2023-08-26T13:55:32.272 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ local cluster_var_name=local_cluster 2023-08-26T13:55:32.273 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ local instance_var_name=instance 2023-08-26T13:55:32.273 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ cluster=cluster1 2023-08-26T13:55:32.273 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ instance=0 2023-08-26T13:55:32.273 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ '[' 0 = cluster1:0 ']' 2023-08-26T13:55:32.273 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ eval local_cluster=cluster1 2023-08-26T13:55:32.274 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++++ local_cluster=cluster1 2023-08-26T13:55:32.274 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ eval instance=0 2023-08-26T13:55:32.274 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++++ instance=0 2023-08-26T13:55:32.274 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2023-08-26T13:55:32.287 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-08-26T13:55:32.288 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-08-26T13:55:32.288 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-08-26T13:55:32.288 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_24 2023-08-26T13:55:32.396 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:Invalid command: unused arguments: ['mirror/image_24'] 2023-08-26T13:55:32.397 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:rbd mirror status : get status for rbd mirror 2023-08-26T13:55:32.397 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:admin_socket: invalid command 2023-08-26T13:55:32.400 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-08-26T13:55:32.400 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_24 2023-08-26T13:55:32.400 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local cluster=cluster1:1 2023-08-26T13:55:32.400 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ shift 2023-08-26T13:55:32.400 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local instance 2023-08-26T13:55:32.401 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ set_cluster_instance cluster1:1 cluster instance 2023-08-26T13:55:32.401 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local val=cluster1:1 2023-08-26T13:55:32.401 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local cluster_var_name=cluster 2023-08-26T13:55:32.401 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local instance_var_name=instance 2023-08-26T13:55:32.401 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ cluster=cluster1 2023-08-26T13:55:32.402 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ instance=1 2023-08-26T13:55:32.402 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ '[' 1 = cluster1:1 ']' 2023-08-26T13:55:32.402 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ eval cluster=cluster1 2023-08-26T13:55:32.402 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ cluster=cluster1 2023-08-26T13:55:32.402 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ eval instance=1 2023-08-26T13:55:32.402 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ instance=1 2023-08-26T13:55:32.403 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ daemon_asok_file cluster1:1 cluster1 2023-08-26T13:55:32.403 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ local local_cluster=cluster1:1 2023-08-26T13:55:32.403 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ local cluster=cluster1 2023-08-26T13:55:32.404 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ local instance 2023-08-26T13:55:32.404 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2023-08-26T13:55:32.404 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ local val=cluster1:1 2023-08-26T13:55:32.404 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ local cluster_var_name=local_cluster 2023-08-26T13:55:32.404 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ local instance_var_name=instance 2023-08-26T13:55:32.405 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ cluster=cluster1 2023-08-26T13:55:32.405 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ instance=1 2023-08-26T13:55:32.405 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ '[' 1 = cluster1:1 ']' 2023-08-26T13:55:32.405 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ eval local_cluster=cluster1 2023-08-26T13:55:32.405 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++++ local_cluster=cluster1 2023-08-26T13:55:32.406 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ eval instance=1 2023-08-26T13:55:32.406 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++++ instance=1 2023-08-26T13:55:32.406 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2023-08-26T13:55:32.418 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-08-26T13:55:32.418 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-08-26T13:55:32.419 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-08-26T13:55:32.419 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_24 2023-08-26T13:55:32.530 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ return 0 2023-08-26T13:55:32.531 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ status_result=' "state": "Replaying"' 2023-08-26T13:55:32.531 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ echo ' "state": "Replaying"' 2023-08-26T13:55:32.532 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ grep -i Replaying 2023-08-26T13:55:32.532 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: "state": "Replaying" 2023-08-26T13:55:32.534 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ current_state=started 2023-08-26T13:55:32.534 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ test started = started 2023-08-26T13:55:32.534 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ return 0 2023-08-26T13:55:32.535 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_24 2023-08-26T13:55:32.535 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local local_cluster=cluster1 2023-08-26T13:55:32.535 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster2 2023-08-26T13:55:32.535 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T13:55:32.535 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=image_24 2023-08-26T13:55:32.536 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ '[' snapshot = journal ']' 2023-08-26T13:55:32.536 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ '[' snapshot = snapshot ']' 2023-08-26T13:55:32.536 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_24 2023-08-26T13:55:32.536 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local local_cluster=cluster1 2023-08-26T13:55:32.536 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster2 2023-08-26T13:55:32.537 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T13:55:32.537 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=image_24 2023-08-26T13:55:32.537 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ mkfname cluster2-mirror-image_24.status 2023-08-26T13:55:32.537 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ echo cluster2-mirror-image_24.status 2023-08-26T13:55:32.537 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ sed -e 's|[/ ]|_|g' 2023-08-26T13:55:32.537 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local status_log=/tmp/tmp.TfDn6BcQ3N/cluster2-mirror-image_24.status 2023-08-26T13:55:32.538 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ mkfname cluster1-mirror-image_24.status 2023-08-26T13:55:32.539 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ echo cluster1-mirror-image_24.status 2023-08-26T13:55:32.539 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ sed -e 's|[/ ]|_|g' 2023-08-26T13:55:32.539 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local local_status_log=/tmp/tmp.TfDn6BcQ3N/cluster1-mirror-image_24.status 2023-08-26T13:55:32.539 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ mirror_image_snapshot cluster2 mirror image_24 2023-08-26T13:55:32.540 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster2 2023-08-26T13:55:32.540 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T13:55:32.540 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=image_24 2023-08-26T13:55:32.540 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_24 2023-08-26T13:55:33.269 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:Snapshot ID: 165 2023-08-26T13:55:33.278 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_24 /tmp/tmp.TfDn6BcQ3N/cluster2-mirror-image_24.status 2023-08-26T13:55:33.279 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster2 2023-08-26T13:55:33.279 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T13:55:33.279 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=image_24 2023-08-26T13:55:33.279 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local log=/tmp/tmp.TfDn6BcQ3N/cluster2-mirror-image_24.status 2023-08-26T13:55:33.280 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_24 --format xml 2023-08-26T13:55:33.280 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-08-26T13:55:33.322 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ xmlstarlet sel -t -v //snapshot/id 2023-08-26T13:55:33.324 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local snapshot_id=165 2023-08-26T13:55:33.324 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ true 2023-08-26T13:55:33.325 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-08-26T13:55:33.325 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ sleep 0.2 2023-08-26T13:55:33.525 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_24 /tmp/tmp.TfDn6BcQ3N/cluster1-mirror-image_24.status 2023-08-26T13:55:33.526 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster1 2023-08-26T13:55:33.526 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T13:55:33.526 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=image_24 2023-08-26T13:55:33.526 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local log=/tmp/tmp.TfDn6BcQ3N/cluster1-mirror-image_24.status 2023-08-26T13:55:33.527 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_24 --format xml 2023-08-26T13:55:33.527 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-08-26T13:55:33.578 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ true 2023-08-26T13:55:33.579 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-08-26T13:55:33.580 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:-:1.1: Document is empty 2023-08-26T13:55:33.580 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr: 2023-08-26T13:55:33.580 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:^ 2023-08-26T13:55:33.581 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local primary_snapshot_id= 2023-08-26T13:55:33.581 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ test 165 = '' 2023-08-26T13:55:33.581 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-08-26T13:55:33.581 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ sleep 0.4 2023-08-26T13:55:33.981 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_24 /tmp/tmp.TfDn6BcQ3N/cluster1-mirror-image_24.status 2023-08-26T13:55:33.982 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster1 2023-08-26T13:55:33.982 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T13:55:33.982 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=image_24 2023-08-26T13:55:33.982 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local log=/tmp/tmp.TfDn6BcQ3N/cluster1-mirror-image_24.status 2023-08-26T13:55:33.982 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_24 --format xml 2023-08-26T13:55:33.983 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-08-26T13:55:34.031 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ true 2023-08-26T13:55:34.032 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-08-26T13:55:34.033 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:-:1.1: Document is empty 2023-08-26T13:55:34.033 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr: 2023-08-26T13:55:34.033 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:^ 2023-08-26T13:55:34.034 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local primary_snapshot_id= 2023-08-26T13:55:34.034 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ test 165 = '' 2023-08-26T13:55:34.034 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-08-26T13:55:34.034 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ sleep 0.8 2023-08-26T13:55:34.835 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_24 /tmp/tmp.TfDn6BcQ3N/cluster1-mirror-image_24.status 2023-08-26T13:55:34.835 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster1 2023-08-26T13:55:34.835 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T13:55:34.836 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=image_24 2023-08-26T13:55:34.836 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local log=/tmp/tmp.TfDn6BcQ3N/cluster1-mirror-image_24.status 2023-08-26T13:55:34.836 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_24 --format xml 2023-08-26T13:55:34.836 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-08-26T13:55:34.882 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-08-26T13:55:34.885 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local primary_snapshot_id=165 2023-08-26T13:55:34.885 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ test 165 = 165 2023-08-26T13:55:34.885 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ return 0 2023-08-26T13:55:34.886 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ wait_for_snap_present cluster1 mirror image_24 snap 2023-08-26T13:55:34.886 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster1 2023-08-26T13:55:34.886 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T13:55:34.886 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=image_24 2023-08-26T13:55:34.886 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local snap_name=snap 2023-08-26T13:55:34.886 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local s 2023-08-26T13:55:34.887 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2023-08-26T13:55:34.887 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ sleep 1 2023-08-26T13:55:35.886 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rbd --cluster cluster1 info mirror/image_24@snap 2023-08-26T13:55:35.933 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:rbd image 'image_24': 2023-08-26T13:55:35.933 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: size 128 MiB in 32 objects 2023-08-26T13:55:35.934 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: order 22 (4 MiB objects) 2023-08-26T13:55:35.934 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: snapshot_count: 2 2023-08-26T13:55:35.934 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: id: 12c04adffadf 2023-08-26T13:55:35.934 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: block_name_prefix: rbd_data.12c04adffadf 2023-08-26T13:55:35.934 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: format: 2 2023-08-26T13:55:35.935 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: features: layering, exclusive-lock, non-primary 2023-08-26T13:55:35.935 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: op_features: 2023-08-26T13:55:35.935 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: flags: 2023-08-26T13:55:35.935 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: create_timestamp: Sat Aug 26 13:46:32 2023 2023-08-26T13:55:35.935 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: access_timestamp: Sat Aug 26 13:46:32 2023 2023-08-26T13:55:35.936 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: modify_timestamp: Sat Aug 26 13:46:32 2023 2023-08-26T13:55:35.936 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: protected: False 2023-08-26T13:55:35.936 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: mirroring state: enabled 2023-08-26T13:55:35.936 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: mirroring mode: snapshot 2023-08-26T13:55:35.936 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: mirroring global id: bfc1dd79-696c-4322-bf38-f1dc8a31a580 2023-08-26T13:55:35.937 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: mirroring primary: false 2023-08-26T13:55:35.938 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ return 0 2023-08-26T13:55:35.938 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ compare_image_snaps mirror image_24 snap 2023-08-26T13:55:35.938 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T13:55:35.938 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=image_24 2023-08-26T13:55:35.939 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local snap_name=snap 2023-08-26T13:55:35.939 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local ret=0 2023-08-26T13:55:35.939 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local rmt_export=/tmp/tmp.TfDn6BcQ3N/cluster2-mirror-image_24.export 2023-08-26T13:55:35.939 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local loc_export=/tmp/tmp.TfDn6BcQ3N/cluster1-mirror-image_24.export 2023-08-26T13:55:35.939 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rm -f /tmp/tmp.TfDn6BcQ3N/cluster2-mirror-image_24.export /tmp/tmp.TfDn6BcQ3N/cluster1-mirror-image_24.export 2023-08-26T13:55:35.940 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rbd --cluster cluster2 -p mirror export image_24@snap /tmp/tmp.TfDn6BcQ3N/cluster2-mirror-image_24.export 2023-08-26T13:55:36.118 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-08-26T13:55:36.125 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rbd --cluster cluster1 -p mirror export image_24@snap /tmp/tmp.TfDn6BcQ3N/cluster1-mirror-image_24.export 2023-08-26T13:55:36.333 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-08-26T13:55:36.339 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ cmp /tmp/tmp.TfDn6BcQ3N/cluster2-mirror-image_24.export /tmp/tmp.TfDn6BcQ3N/cluster1-mirror-image_24.export 2023-08-26T13:55:36.394 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rm -f /tmp/tmp.TfDn6BcQ3N/cluster2-mirror-image_24.export /tmp/tmp.TfDn6BcQ3N/cluster1-mirror-image_24.export 2023-08-26T13:55:36.432 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ return 0 2023-08-26T13:55:36.432 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-08-26T13:55:36.433 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ image=image_25 2023-08-26T13:55:36.433 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ create_snap cluster2 mirror image_25 snap 2023-08-26T13:55:36.433 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster2 2023-08-26T13:55:36.433 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T13:55:36.433 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=image_25 2023-08-26T13:55:36.434 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local snap_name=snap 2023-08-26T13:55:36.434 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rbd --cluster cluster2 -p mirror snap create image_25@snap --debug-rbd=20 --debug-journaler=20 2023-08-26T13:55:38.289 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ wait_for_image_replay_started cluster1 mirror image_25 2023-08-26T13:55:38.289 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster1 2023-08-26T13:55:38.290 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T13:55:38.290 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=image_25 2023-08-26T13:55:38.290 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ wait_for_image_replay_state cluster1 mirror image_25 started 2023-08-26T13:55:38.290 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster1 2023-08-26T13:55:38.290 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T13:55:38.291 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=image_25 2023-08-26T13:55:38.291 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local state=started 2023-08-26T13:55:38.291 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local s 2023-08-26T13:55:38.291 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-08-26T13:55:38.291 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ sleep 1 2023-08-26T13:55:39.292 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ test_image_replay_state cluster1 mirror image_25 started 2023-08-26T13:55:39.292 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster1 2023-08-26T13:55:39.293 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T13:55:39.293 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=image_25 2023-08-26T13:55:39.293 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local test_state=started 2023-08-26T13:55:39.294 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local status_result 2023-08-26T13:55:39.294 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local current_state=stopped 2023-08-26T13:55:39.294 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_25 2023-08-26T13:55:39.295 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ grep -i state 2023-08-26T13:55:39.295 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local cluster_instance=cluster1 2023-08-26T13:55:39.295 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ shift 2023-08-26T13:55:39.295 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local cluster=cluster1 2023-08-26T13:55:39.296 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local instance=cluster1 2023-08-26T13:55:39.296 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local loop_instance 2023-08-26T13:55:39.296 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-08-26T13:55:39.297 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ sleep 0 2023-08-26T13:55:39.297 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ '[' cluster1 '!=' cluster1 ']' 2023-08-26T13:55:39.297 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ seq 0 3 2023-08-26T13:55:39.301 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-08-26T13:55:39.302 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_25 2023-08-26T13:55:39.302 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local cluster=cluster1:0 2023-08-26T13:55:39.302 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ shift 2023-08-26T13:55:39.303 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local instance 2023-08-26T13:55:39.303 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ set_cluster_instance cluster1:0 cluster instance 2023-08-26T13:55:39.303 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local val=cluster1:0 2023-08-26T13:55:39.303 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local cluster_var_name=cluster 2023-08-26T13:55:39.304 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local instance_var_name=instance 2023-08-26T13:55:39.304 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ cluster=cluster1 2023-08-26T13:55:39.304 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ instance=0 2023-08-26T13:55:39.305 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ '[' 0 = cluster1:0 ']' 2023-08-26T13:55:39.305 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ eval cluster=cluster1 2023-08-26T13:55:39.305 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ cluster=cluster1 2023-08-26T13:55:39.305 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ eval instance=0 2023-08-26T13:55:39.306 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ instance=0 2023-08-26T13:55:39.306 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ daemon_asok_file cluster1:0 cluster1 2023-08-26T13:55:39.306 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ local local_cluster=cluster1:0 2023-08-26T13:55:39.307 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ local cluster=cluster1 2023-08-26T13:55:39.307 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ local instance 2023-08-26T13:55:39.307 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2023-08-26T13:55:39.307 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ local val=cluster1:0 2023-08-26T13:55:39.308 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ local cluster_var_name=local_cluster 2023-08-26T13:55:39.308 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ local instance_var_name=instance 2023-08-26T13:55:39.308 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ cluster=cluster1 2023-08-26T13:55:39.308 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ instance=0 2023-08-26T13:55:39.309 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ '[' 0 = cluster1:0 ']' 2023-08-26T13:55:39.309 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ eval local_cluster=cluster1 2023-08-26T13:55:39.309 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++++ local_cluster=cluster1 2023-08-26T13:55:39.310 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ eval instance=0 2023-08-26T13:55:39.310 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++++ instance=0 2023-08-26T13:55:39.310 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2023-08-26T13:55:39.317 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-08-26T13:55:39.317 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-08-26T13:55:39.317 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-08-26T13:55:39.317 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_25 2023-08-26T13:55:39.425 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:Invalid command: unused arguments: ['mirror/image_25'] 2023-08-26T13:55:39.425 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:rbd mirror status : get status for rbd mirror 2023-08-26T13:55:39.425 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:admin_socket: invalid command 2023-08-26T13:55:39.428 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-08-26T13:55:39.428 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_25 2023-08-26T13:55:39.429 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local cluster=cluster1:1 2023-08-26T13:55:39.429 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ shift 2023-08-26T13:55:39.429 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local instance 2023-08-26T13:55:39.429 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ set_cluster_instance cluster1:1 cluster instance 2023-08-26T13:55:39.429 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local val=cluster1:1 2023-08-26T13:55:39.430 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local cluster_var_name=cluster 2023-08-26T13:55:39.430 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local instance_var_name=instance 2023-08-26T13:55:39.430 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ cluster=cluster1 2023-08-26T13:55:39.430 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ instance=1 2023-08-26T13:55:39.430 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ '[' 1 = cluster1:1 ']' 2023-08-26T13:55:39.431 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ eval cluster=cluster1 2023-08-26T13:55:39.431 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ cluster=cluster1 2023-08-26T13:55:39.431 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ eval instance=1 2023-08-26T13:55:39.431 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ instance=1 2023-08-26T13:55:39.431 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ daemon_asok_file cluster1:1 cluster1 2023-08-26T13:55:39.431 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ local local_cluster=cluster1:1 2023-08-26T13:55:39.432 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ local cluster=cluster1 2023-08-26T13:55:39.432 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ local instance 2023-08-26T13:55:39.432 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2023-08-26T13:55:39.432 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ local val=cluster1:1 2023-08-26T13:55:39.432 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ local cluster_var_name=local_cluster 2023-08-26T13:55:39.432 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ local instance_var_name=instance 2023-08-26T13:55:39.433 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ cluster=cluster1 2023-08-26T13:55:39.433 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ instance=1 2023-08-26T13:55:39.433 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ '[' 1 = cluster1:1 ']' 2023-08-26T13:55:39.433 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ eval local_cluster=cluster1 2023-08-26T13:55:39.433 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++++ local_cluster=cluster1 2023-08-26T13:55:39.433 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ eval instance=1 2023-08-26T13:55:39.434 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++++ instance=1 2023-08-26T13:55:39.434 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2023-08-26T13:55:39.446 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-08-26T13:55:39.447 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-08-26T13:55:39.447 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-08-26T13:55:39.447 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_25 2023-08-26T13:55:39.558 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ return 0 2023-08-26T13:55:39.559 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ status_result=' "state": "Replaying"' 2023-08-26T13:55:39.560 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ echo ' "state": "Replaying"' 2023-08-26T13:55:39.560 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ grep -i Replaying 2023-08-26T13:55:39.560 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: "state": "Replaying" 2023-08-26T13:55:39.562 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ current_state=started 2023-08-26T13:55:39.562 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ test started = started 2023-08-26T13:55:39.562 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ return 0 2023-08-26T13:55:39.563 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_25 2023-08-26T13:55:39.563 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local local_cluster=cluster1 2023-08-26T13:55:39.563 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster2 2023-08-26T13:55:39.563 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T13:55:39.563 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=image_25 2023-08-26T13:55:39.564 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ '[' snapshot = journal ']' 2023-08-26T13:55:39.564 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ '[' snapshot = snapshot ']' 2023-08-26T13:55:39.564 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_25 2023-08-26T13:55:39.564 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local local_cluster=cluster1 2023-08-26T13:55:39.564 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster2 2023-08-26T13:55:39.564 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T13:55:39.565 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=image_25 2023-08-26T13:55:39.565 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ mkfname cluster2-mirror-image_25.status 2023-08-26T13:55:39.565 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ echo cluster2-mirror-image_25.status 2023-08-26T13:55:39.565 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ sed -e 's|[/ ]|_|g' 2023-08-26T13:55:39.565 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local status_log=/tmp/tmp.TfDn6BcQ3N/cluster2-mirror-image_25.status 2023-08-26T13:55:39.566 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ mkfname cluster1-mirror-image_25.status 2023-08-26T13:55:39.567 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ echo cluster1-mirror-image_25.status 2023-08-26T13:55:39.567 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ sed -e 's|[/ ]|_|g' 2023-08-26T13:55:39.567 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local local_status_log=/tmp/tmp.TfDn6BcQ3N/cluster1-mirror-image_25.status 2023-08-26T13:55:39.567 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ mirror_image_snapshot cluster2 mirror image_25 2023-08-26T13:55:39.568 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster2 2023-08-26T13:55:39.568 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T13:55:39.568 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=image_25 2023-08-26T13:55:39.568 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_25 2023-08-26T13:55:40.290 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:Snapshot ID: 167 2023-08-26T13:55:40.299 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_25 /tmp/tmp.TfDn6BcQ3N/cluster2-mirror-image_25.status 2023-08-26T13:55:40.300 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster2 2023-08-26T13:55:40.300 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T13:55:40.300 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=image_25 2023-08-26T13:55:40.301 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local log=/tmp/tmp.TfDn6BcQ3N/cluster2-mirror-image_25.status 2023-08-26T13:55:40.301 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_25 --format xml 2023-08-26T13:55:40.301 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-08-26T13:55:40.342 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ xmlstarlet sel -t -v //snapshot/id 2023-08-26T13:55:40.344 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local snapshot_id=167 2023-08-26T13:55:40.344 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ true 2023-08-26T13:55:40.344 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-08-26T13:55:40.345 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ sleep 0.2 2023-08-26T13:55:40.545 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_25 /tmp/tmp.TfDn6BcQ3N/cluster1-mirror-image_25.status 2023-08-26T13:55:40.546 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster1 2023-08-26T13:55:40.546 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T13:55:40.546 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=image_25 2023-08-26T13:55:40.546 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local log=/tmp/tmp.TfDn6BcQ3N/cluster1-mirror-image_25.status 2023-08-26T13:55:40.546 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_25 --format xml 2023-08-26T13:55:40.546 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-08-26T13:55:40.592 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-08-26T13:55:40.593 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local primary_snapshot_id=92 2023-08-26T13:55:40.594 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ test 167 = 92 2023-08-26T13:55:40.594 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-08-26T13:55:40.594 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ sleep 0.4 2023-08-26T13:55:40.995 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_25 /tmp/tmp.TfDn6BcQ3N/cluster1-mirror-image_25.status 2023-08-26T13:55:40.995 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster1 2023-08-26T13:55:40.996 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T13:55:40.996 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=image_25 2023-08-26T13:55:40.996 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local log=/tmp/tmp.TfDn6BcQ3N/cluster1-mirror-image_25.status 2023-08-26T13:55:40.996 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_25 --format xml 2023-08-26T13:55:40.996 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-08-26T13:55:41.043 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-08-26T13:55:41.045 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local primary_snapshot_id=92 2023-08-26T13:55:41.045 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ test 167 = 92 2023-08-26T13:55:41.045 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-08-26T13:55:41.045 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ sleep 0.8 2023-08-26T13:55:41.846 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_25 /tmp/tmp.TfDn6BcQ3N/cluster1-mirror-image_25.status 2023-08-26T13:55:41.847 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster1 2023-08-26T13:55:41.847 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T13:55:41.847 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=image_25 2023-08-26T13:55:41.847 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local log=/tmp/tmp.TfDn6BcQ3N/cluster1-mirror-image_25.status 2023-08-26T13:55:41.847 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_25 --format xml 2023-08-26T13:55:41.847 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-08-26T13:55:41.897 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ true 2023-08-26T13:55:41.898 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-08-26T13:55:41.899 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:-:1.1: Document is empty 2023-08-26T13:55:41.899 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr: 2023-08-26T13:55:41.899 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:^ 2023-08-26T13:55:41.900 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local primary_snapshot_id= 2023-08-26T13:55:41.900 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ test 167 = '' 2023-08-26T13:55:41.900 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-08-26T13:55:41.900 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ sleep 1.6 2023-08-26T13:55:43.501 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_25 /tmp/tmp.TfDn6BcQ3N/cluster1-mirror-image_25.status 2023-08-26T13:55:43.501 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster1 2023-08-26T13:55:43.501 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T13:55:43.501 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=image_25 2023-08-26T13:55:43.501 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local log=/tmp/tmp.TfDn6BcQ3N/cluster1-mirror-image_25.status 2023-08-26T13:55:43.502 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_25 --format xml 2023-08-26T13:55:43.502 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-08-26T13:55:43.548 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-08-26T13:55:43.550 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local primary_snapshot_id=167 2023-08-26T13:55:43.550 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ test 167 = 167 2023-08-26T13:55:43.551 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ return 0 2023-08-26T13:55:43.551 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ wait_for_snap_present cluster1 mirror image_25 snap 2023-08-26T13:55:43.551 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster1 2023-08-26T13:55:43.551 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T13:55:43.551 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=image_25 2023-08-26T13:55:43.551 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local snap_name=snap 2023-08-26T13:55:43.552 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local s 2023-08-26T13:55:43.552 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2023-08-26T13:55:43.552 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ sleep 1 2023-08-26T13:55:44.551 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rbd --cluster cluster1 info mirror/image_25@snap 2023-08-26T13:55:44.595 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:rbd image 'image_25': 2023-08-26T13:55:44.596 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: size 128 MiB in 32 objects 2023-08-26T13:55:44.596 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: order 22 (4 MiB objects) 2023-08-26T13:55:44.596 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: snapshot_count: 2 2023-08-26T13:55:44.596 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: id: 10b76593d29c 2023-08-26T13:55:44.597 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: block_name_prefix: rbd_data.10b76593d29c 2023-08-26T13:55:44.597 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: format: 2 2023-08-26T13:55:44.597 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: features: layering, exclusive-lock, non-primary 2023-08-26T13:55:44.597 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: op_features: 2023-08-26T13:55:44.597 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: flags: 2023-08-26T13:55:44.598 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: create_timestamp: Sat Aug 26 13:46:46 2023 2023-08-26T13:55:44.598 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: access_timestamp: Sat Aug 26 13:46:46 2023 2023-08-26T13:55:44.598 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: modify_timestamp: Sat Aug 26 13:46:46 2023 2023-08-26T13:55:44.599 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: protected: False 2023-08-26T13:55:44.599 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: mirroring state: enabled 2023-08-26T13:55:44.599 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: mirroring mode: snapshot 2023-08-26T13:55:44.599 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: mirroring global id: ae7a1e38-d406-47df-97d4-9cbf52592bb7 2023-08-26T13:55:44.599 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: mirroring primary: false 2023-08-26T13:55:44.601 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ return 0 2023-08-26T13:55:44.601 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ compare_image_snaps mirror image_25 snap 2023-08-26T13:55:44.601 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T13:55:44.601 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=image_25 2023-08-26T13:55:44.601 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local snap_name=snap 2023-08-26T13:55:44.602 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local ret=0 2023-08-26T13:55:44.602 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local rmt_export=/tmp/tmp.TfDn6BcQ3N/cluster2-mirror-image_25.export 2023-08-26T13:55:44.602 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local loc_export=/tmp/tmp.TfDn6BcQ3N/cluster1-mirror-image_25.export 2023-08-26T13:55:44.602 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rm -f /tmp/tmp.TfDn6BcQ3N/cluster2-mirror-image_25.export /tmp/tmp.TfDn6BcQ3N/cluster1-mirror-image_25.export 2023-08-26T13:55:44.602 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rbd --cluster cluster2 -p mirror export image_25@snap /tmp/tmp.TfDn6BcQ3N/cluster2-mirror-image_25.export 2023-08-26T13:55:44.778 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-08-26T13:55:44.784 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rbd --cluster cluster1 -p mirror export image_25@snap /tmp/tmp.TfDn6BcQ3N/cluster1-mirror-image_25.export 2023-08-26T13:55:45.222 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-08-26T13:55:45.229 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ cmp /tmp/tmp.TfDn6BcQ3N/cluster2-mirror-image_25.export /tmp/tmp.TfDn6BcQ3N/cluster1-mirror-image_25.export 2023-08-26T13:55:45.284 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rm -f /tmp/tmp.TfDn6BcQ3N/cluster2-mirror-image_25.export /tmp/tmp.TfDn6BcQ3N/cluster1-mirror-image_25.export 2023-08-26T13:55:45.322 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ return 0 2023-08-26T13:55:45.322 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-08-26T13:55:45.322 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ image=image_26 2023-08-26T13:55:45.323 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ create_snap cluster2 mirror image_26 snap 2023-08-26T13:55:45.323 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster2 2023-08-26T13:55:45.323 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T13:55:45.323 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=image_26 2023-08-26T13:55:45.323 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local snap_name=snap 2023-08-26T13:55:45.324 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rbd --cluster cluster2 -p mirror snap create image_26@snap --debug-rbd=20 --debug-journaler=20 2023-08-26T13:55:47.187 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ wait_for_image_replay_started cluster1 mirror image_26 2023-08-26T13:55:47.188 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster1 2023-08-26T13:55:47.188 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T13:55:47.188 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=image_26 2023-08-26T13:55:47.188 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ wait_for_image_replay_state cluster1 mirror image_26 started 2023-08-26T13:55:47.188 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster1 2023-08-26T13:55:47.189 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T13:55:47.189 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=image_26 2023-08-26T13:55:47.189 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local state=started 2023-08-26T13:55:47.189 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local s 2023-08-26T13:55:47.189 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-08-26T13:55:47.189 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ sleep 1 2023-08-26T13:55:48.188 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ test_image_replay_state cluster1 mirror image_26 started 2023-08-26T13:55:48.189 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster1 2023-08-26T13:55:48.189 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T13:55:48.189 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=image_26 2023-08-26T13:55:48.189 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local test_state=started 2023-08-26T13:55:48.190 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local status_result 2023-08-26T13:55:48.190 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local current_state=stopped 2023-08-26T13:55:48.191 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_26 2023-08-26T13:55:48.191 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local cluster_instance=cluster1 2023-08-26T13:55:48.192 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ grep -i state 2023-08-26T13:55:48.192 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ shift 2023-08-26T13:55:48.192 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local cluster=cluster1 2023-08-26T13:55:48.192 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local instance=cluster1 2023-08-26T13:55:48.192 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local loop_instance 2023-08-26T13:55:48.193 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-08-26T13:55:48.193 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ sleep 0 2023-08-26T13:55:48.193 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ '[' cluster1 '!=' cluster1 ']' 2023-08-26T13:55:48.193 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ seq 0 3 2023-08-26T13:55:48.195 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-08-26T13:55:48.195 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_26 2023-08-26T13:55:48.195 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local cluster=cluster1:0 2023-08-26T13:55:48.196 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ shift 2023-08-26T13:55:48.196 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local instance 2023-08-26T13:55:48.196 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ set_cluster_instance cluster1:0 cluster instance 2023-08-26T13:55:48.196 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local val=cluster1:0 2023-08-26T13:55:48.196 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local cluster_var_name=cluster 2023-08-26T13:55:48.197 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local instance_var_name=instance 2023-08-26T13:55:48.197 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ cluster=cluster1 2023-08-26T13:55:48.197 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ instance=0 2023-08-26T13:55:48.197 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ '[' 0 = cluster1:0 ']' 2023-08-26T13:55:48.197 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ eval cluster=cluster1 2023-08-26T13:55:48.198 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ cluster=cluster1 2023-08-26T13:55:48.198 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ eval instance=0 2023-08-26T13:55:48.198 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ instance=0 2023-08-26T13:55:48.199 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ daemon_asok_file cluster1:0 cluster1 2023-08-26T13:55:48.200 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ local local_cluster=cluster1:0 2023-08-26T13:55:48.200 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ local cluster=cluster1 2023-08-26T13:55:48.200 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ local instance 2023-08-26T13:55:48.200 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2023-08-26T13:55:48.201 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ local val=cluster1:0 2023-08-26T13:55:48.201 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ local cluster_var_name=local_cluster 2023-08-26T13:55:48.201 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ local instance_var_name=instance 2023-08-26T13:55:48.201 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ cluster=cluster1 2023-08-26T13:55:48.201 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ instance=0 2023-08-26T13:55:48.201 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ '[' 0 = cluster1:0 ']' 2023-08-26T13:55:48.202 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ eval local_cluster=cluster1 2023-08-26T13:55:48.202 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++++ local_cluster=cluster1 2023-08-26T13:55:48.202 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ eval instance=0 2023-08-26T13:55:48.202 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++++ instance=0 2023-08-26T13:55:48.202 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2023-08-26T13:55:48.216 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-08-26T13:55:48.216 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-08-26T13:55:48.217 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-08-26T13:55:48.217 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_26 2023-08-26T13:55:48.338 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ return 0 2023-08-26T13:55:48.339 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ status_result=' "state": "Replaying"' 2023-08-26T13:55:48.339 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ echo ' "state": "Replaying"' 2023-08-26T13:55:48.339 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ grep -i Replaying 2023-08-26T13:55:48.340 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: "state": "Replaying" 2023-08-26T13:55:48.342 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ current_state=started 2023-08-26T13:55:48.342 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ test started = started 2023-08-26T13:55:48.342 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ return 0 2023-08-26T13:55:48.343 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_26 2023-08-26T13:55:48.343 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local local_cluster=cluster1 2023-08-26T13:55:48.343 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster2 2023-08-26T13:55:48.343 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T13:55:48.344 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=image_26 2023-08-26T13:55:48.344 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ '[' snapshot = journal ']' 2023-08-26T13:55:48.344 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ '[' snapshot = snapshot ']' 2023-08-26T13:55:48.344 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_26 2023-08-26T13:55:48.344 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local local_cluster=cluster1 2023-08-26T13:55:48.344 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster2 2023-08-26T13:55:48.345 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T13:55:48.345 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=image_26 2023-08-26T13:55:48.345 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ mkfname cluster2-mirror-image_26.status 2023-08-26T13:55:48.345 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ echo cluster2-mirror-image_26.status 2023-08-26T13:55:48.345 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ sed -e 's|[/ ]|_|g' 2023-08-26T13:55:48.346 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local status_log=/tmp/tmp.TfDn6BcQ3N/cluster2-mirror-image_26.status 2023-08-26T13:55:48.346 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ mkfname cluster1-mirror-image_26.status 2023-08-26T13:55:48.346 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ echo cluster1-mirror-image_26.status 2023-08-26T13:55:48.346 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ sed -e 's|[/ ]|_|g' 2023-08-26T13:55:48.347 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local local_status_log=/tmp/tmp.TfDn6BcQ3N/cluster1-mirror-image_26.status 2023-08-26T13:55:48.347 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ mirror_image_snapshot cluster2 mirror image_26 2023-08-26T13:55:48.348 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster2 2023-08-26T13:55:48.348 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T13:55:48.348 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=image_26 2023-08-26T13:55:48.348 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_26 2023-08-26T13:55:49.201 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:Snapshot ID: 169 2023-08-26T13:55:49.210 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_26 /tmp/tmp.TfDn6BcQ3N/cluster2-mirror-image_26.status 2023-08-26T13:55:49.210 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster2 2023-08-26T13:55:49.210 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T13:55:49.211 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=image_26 2023-08-26T13:55:49.211 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local log=/tmp/tmp.TfDn6BcQ3N/cluster2-mirror-image_26.status 2023-08-26T13:55:49.211 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_26 --format xml 2023-08-26T13:55:49.211 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-08-26T13:55:49.252 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ xmlstarlet sel -t -v //snapshot/id 2023-08-26T13:55:49.254 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local snapshot_id=169 2023-08-26T13:55:49.254 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ true 2023-08-26T13:55:49.255 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-08-26T13:55:49.255 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ sleep 0.2 2023-08-26T13:55:49.455 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_26 /tmp/tmp.TfDn6BcQ3N/cluster1-mirror-image_26.status 2023-08-26T13:55:49.456 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster1 2023-08-26T13:55:49.456 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T13:55:49.456 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=image_26 2023-08-26T13:55:49.456 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local log=/tmp/tmp.TfDn6BcQ3N/cluster1-mirror-image_26.status 2023-08-26T13:55:49.456 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_26 --format xml 2023-08-26T13:55:49.457 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-08-26T13:55:49.503 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-08-26T13:55:49.505 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local primary_snapshot_id=93 2023-08-26T13:55:49.505 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ test 169 = 93 2023-08-26T13:55:49.505 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-08-26T13:55:49.505 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ sleep 0.4 2023-08-26T13:55:49.906 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_26 /tmp/tmp.TfDn6BcQ3N/cluster1-mirror-image_26.status 2023-08-26T13:55:49.907 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster1 2023-08-26T13:55:49.907 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T13:55:49.907 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=image_26 2023-08-26T13:55:49.907 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local log=/tmp/tmp.TfDn6BcQ3N/cluster1-mirror-image_26.status 2023-08-26T13:55:49.907 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_26 --format xml 2023-08-26T13:55:49.908 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-08-26T13:55:49.956 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ true 2023-08-26T13:55:49.956 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-08-26T13:55:49.957 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:-:1.1: Document is empty 2023-08-26T13:55:49.958 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr: 2023-08-26T13:55:49.958 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:^ 2023-08-26T13:55:49.958 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local primary_snapshot_id= 2023-08-26T13:55:49.959 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ test 169 = '' 2023-08-26T13:55:49.959 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-08-26T13:55:49.959 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ sleep 0.8 2023-08-26T13:55:50.759 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_26 /tmp/tmp.TfDn6BcQ3N/cluster1-mirror-image_26.status 2023-08-26T13:55:50.760 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster1 2023-08-26T13:55:50.760 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T13:55:50.760 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=image_26 2023-08-26T13:55:50.760 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local log=/tmp/tmp.TfDn6BcQ3N/cluster1-mirror-image_26.status 2023-08-26T13:55:50.761 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_26 --format xml 2023-08-26T13:55:50.761 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-08-26T13:55:50.804 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-08-26T13:55:50.806 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local primary_snapshot_id=169 2023-08-26T13:55:50.806 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ test 169 = 169 2023-08-26T13:55:50.806 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ return 0 2023-08-26T13:55:50.807 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ wait_for_snap_present cluster1 mirror image_26 snap 2023-08-26T13:55:50.807 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster1 2023-08-26T13:55:50.807 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T13:55:50.808 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=image_26 2023-08-26T13:55:50.808 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local snap_name=snap 2023-08-26T13:55:50.808 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local s 2023-08-26T13:55:50.808 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2023-08-26T13:55:50.808 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ sleep 1 2023-08-26T13:55:51.807 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rbd --cluster cluster1 info mirror/image_26@snap 2023-08-26T13:55:51.849 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:rbd image 'image_26': 2023-08-26T13:55:51.850 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: size 128 MiB in 32 objects 2023-08-26T13:55:51.850 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: order 22 (4 MiB objects) 2023-08-26T13:55:51.850 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: snapshot_count: 2 2023-08-26T13:55:51.851 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: id: 122b92ee6d90 2023-08-26T13:55:51.851 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: block_name_prefix: rbd_data.122b92ee6d90 2023-08-26T13:55:51.851 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: format: 2 2023-08-26T13:55:51.851 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: features: layering, exclusive-lock, non-primary 2023-08-26T13:55:51.851 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: op_features: 2023-08-26T13:55:51.852 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: flags: 2023-08-26T13:55:51.852 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: create_timestamp: Sat Aug 26 13:46:56 2023 2023-08-26T13:55:51.852 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: access_timestamp: Sat Aug 26 13:46:56 2023 2023-08-26T13:55:51.852 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: modify_timestamp: Sat Aug 26 13:46:56 2023 2023-08-26T13:55:51.852 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: protected: False 2023-08-26T13:55:51.853 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: mirroring state: enabled 2023-08-26T13:55:51.853 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: mirroring mode: snapshot 2023-08-26T13:55:51.853 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: mirroring global id: ef7edecb-ca31-4518-87d6-58c3dfe520f7 2023-08-26T13:55:51.853 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: mirroring primary: false 2023-08-26T13:55:51.854 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ return 0 2023-08-26T13:55:51.855 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ compare_image_snaps mirror image_26 snap 2023-08-26T13:55:51.855 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T13:55:51.855 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=image_26 2023-08-26T13:55:51.855 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local snap_name=snap 2023-08-26T13:55:51.856 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local ret=0 2023-08-26T13:55:51.856 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local rmt_export=/tmp/tmp.TfDn6BcQ3N/cluster2-mirror-image_26.export 2023-08-26T13:55:51.856 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local loc_export=/tmp/tmp.TfDn6BcQ3N/cluster1-mirror-image_26.export 2023-08-26T13:55:51.856 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rm -f /tmp/tmp.TfDn6BcQ3N/cluster2-mirror-image_26.export /tmp/tmp.TfDn6BcQ3N/cluster1-mirror-image_26.export 2023-08-26T13:55:51.856 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rbd --cluster cluster2 -p mirror export image_26@snap /tmp/tmp.TfDn6BcQ3N/cluster2-mirror-image_26.export 2023-08-26T13:55:52.042 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-08-26T13:55:52.048 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rbd --cluster cluster1 -p mirror export image_26@snap /tmp/tmp.TfDn6BcQ3N/cluster1-mirror-image_26.export 2023-08-26T13:55:52.283 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-08-26T13:55:52.288 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ cmp /tmp/tmp.TfDn6BcQ3N/cluster2-mirror-image_26.export /tmp/tmp.TfDn6BcQ3N/cluster1-mirror-image_26.export 2023-08-26T13:55:52.342 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rm -f /tmp/tmp.TfDn6BcQ3N/cluster2-mirror-image_26.export /tmp/tmp.TfDn6BcQ3N/cluster1-mirror-image_26.export 2023-08-26T13:55:52.379 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ return 0 2023-08-26T13:55:52.380 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-08-26T13:55:52.380 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ image=image_27 2023-08-26T13:55:52.380 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ create_snap cluster2 mirror image_27 snap 2023-08-26T13:55:52.380 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster2 2023-08-26T13:55:52.380 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T13:55:52.380 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=image_27 2023-08-26T13:55:52.381 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local snap_name=snap 2023-08-26T13:55:52.381 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rbd --cluster cluster2 -p mirror snap create image_27@snap --debug-rbd=20 --debug-journaler=20 2023-08-26T13:55:54.204 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ wait_for_image_replay_started cluster1 mirror image_27 2023-08-26T13:55:54.204 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster1 2023-08-26T13:55:54.204 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T13:55:54.205 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=image_27 2023-08-26T13:55:54.205 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ wait_for_image_replay_state cluster1 mirror image_27 started 2023-08-26T13:55:54.205 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster1 2023-08-26T13:55:54.205 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T13:55:54.205 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=image_27 2023-08-26T13:55:54.206 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local state=started 2023-08-26T13:55:54.206 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local s 2023-08-26T13:55:54.206 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-08-26T13:55:54.206 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ sleep 1 2023-08-26T13:55:55.205 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ test_image_replay_state cluster1 mirror image_27 started 2023-08-26T13:55:55.206 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster1 2023-08-26T13:55:55.207 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T13:55:55.207 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=image_27 2023-08-26T13:55:55.207 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local test_state=started 2023-08-26T13:55:55.207 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local status_result 2023-08-26T13:55:55.208 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local current_state=stopped 2023-08-26T13:55:55.208 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_27 2023-08-26T13:55:55.208 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ grep -i state 2023-08-26T13:55:55.208 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local cluster_instance=cluster1 2023-08-26T13:55:55.208 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ shift 2023-08-26T13:55:55.208 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local cluster=cluster1 2023-08-26T13:55:55.209 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local instance=cluster1 2023-08-26T13:55:55.209 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local loop_instance 2023-08-26T13:55:55.209 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-08-26T13:55:55.209 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ sleep 0 2023-08-26T13:55:55.209 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ '[' cluster1 '!=' cluster1 ']' 2023-08-26T13:55:55.212 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ seq 0 3 2023-08-26T13:55:55.213 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-08-26T13:55:55.213 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_27 2023-08-26T13:55:55.213 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local cluster=cluster1:0 2023-08-26T13:55:55.213 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ shift 2023-08-26T13:55:55.213 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local instance 2023-08-26T13:55:55.214 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ set_cluster_instance cluster1:0 cluster instance 2023-08-26T13:55:55.214 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local val=cluster1:0 2023-08-26T13:55:55.214 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local cluster_var_name=cluster 2023-08-26T13:55:55.214 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local instance_var_name=instance 2023-08-26T13:55:55.214 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ cluster=cluster1 2023-08-26T13:55:55.214 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ instance=0 2023-08-26T13:55:55.215 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ '[' 0 = cluster1:0 ']' 2023-08-26T13:55:55.215 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ eval cluster=cluster1 2023-08-26T13:55:55.215 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ cluster=cluster1 2023-08-26T13:55:55.215 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ eval instance=0 2023-08-26T13:55:55.216 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ instance=0 2023-08-26T13:55:55.216 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ daemon_asok_file cluster1:0 cluster1 2023-08-26T13:55:55.216 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ local local_cluster=cluster1:0 2023-08-26T13:55:55.216 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ local cluster=cluster1 2023-08-26T13:55:55.216 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ local instance 2023-08-26T13:55:55.216 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2023-08-26T13:55:55.217 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ local val=cluster1:0 2023-08-26T13:55:55.217 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ local cluster_var_name=local_cluster 2023-08-26T13:55:55.217 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ local instance_var_name=instance 2023-08-26T13:55:55.217 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ cluster=cluster1 2023-08-26T13:55:55.217 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ instance=0 2023-08-26T13:55:55.218 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ '[' 0 = cluster1:0 ']' 2023-08-26T13:55:55.218 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ eval local_cluster=cluster1 2023-08-26T13:55:55.218 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++++ local_cluster=cluster1 2023-08-26T13:55:55.218 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ eval instance=0 2023-08-26T13:55:55.218 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++++ instance=0 2023-08-26T13:55:55.219 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2023-08-26T13:55:55.231 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-08-26T13:55:55.231 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-08-26T13:55:55.232 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-08-26T13:55:55.232 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_27 2023-08-26T13:55:55.351 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ return 0 2023-08-26T13:55:55.352 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ status_result=' "state": "Replaying"' 2023-08-26T13:55:55.352 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ echo ' "state": "Replaying"' 2023-08-26T13:55:55.352 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ grep -i Replaying 2023-08-26T13:55:55.353 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: "state": "Replaying" 2023-08-26T13:55:55.355 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ current_state=started 2023-08-26T13:55:55.355 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ test started = started 2023-08-26T13:55:55.355 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ return 0 2023-08-26T13:55:55.355 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_27 2023-08-26T13:55:55.356 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local local_cluster=cluster1 2023-08-26T13:55:55.356 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster2 2023-08-26T13:55:55.356 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T13:55:55.356 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=image_27 2023-08-26T13:55:55.356 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ '[' snapshot = journal ']' 2023-08-26T13:55:55.356 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ '[' snapshot = snapshot ']' 2023-08-26T13:55:55.357 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_27 2023-08-26T13:55:55.357 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local local_cluster=cluster1 2023-08-26T13:55:55.357 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster2 2023-08-26T13:55:55.357 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T13:55:55.357 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=image_27 2023-08-26T13:55:55.358 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ mkfname cluster2-mirror-image_27.status 2023-08-26T13:55:55.358 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ echo cluster2-mirror-image_27.status 2023-08-26T13:55:55.358 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ sed -e 's|[/ ]|_|g' 2023-08-26T13:55:55.358 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local status_log=/tmp/tmp.TfDn6BcQ3N/cluster2-mirror-image_27.status 2023-08-26T13:55:55.359 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ mkfname cluster1-mirror-image_27.status 2023-08-26T13:55:55.359 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ echo cluster1-mirror-image_27.status 2023-08-26T13:55:55.360 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ sed -e 's|[/ ]|_|g' 2023-08-26T13:55:55.360 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local local_status_log=/tmp/tmp.TfDn6BcQ3N/cluster1-mirror-image_27.status 2023-08-26T13:55:55.360 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ mirror_image_snapshot cluster2 mirror image_27 2023-08-26T13:55:55.360 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster2 2023-08-26T13:55:55.360 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T13:55:55.361 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=image_27 2023-08-26T13:55:55.361 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_27 2023-08-26T13:55:56.208 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:Snapshot ID: 171 2023-08-26T13:55:56.216 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_27 /tmp/tmp.TfDn6BcQ3N/cluster2-mirror-image_27.status 2023-08-26T13:55:56.216 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster2 2023-08-26T13:55:56.217 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T13:55:56.217 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=image_27 2023-08-26T13:55:56.217 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local log=/tmp/tmp.TfDn6BcQ3N/cluster2-mirror-image_27.status 2023-08-26T13:55:56.217 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_27 --format xml 2023-08-26T13:55:56.217 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-08-26T13:55:56.260 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ xmlstarlet sel -t -v //snapshot/id 2023-08-26T13:55:56.262 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local snapshot_id=171 2023-08-26T13:55:56.262 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ true 2023-08-26T13:55:56.262 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-08-26T13:55:56.263 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ sleep 0.2 2023-08-26T13:55:56.463 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_27 /tmp/tmp.TfDn6BcQ3N/cluster1-mirror-image_27.status 2023-08-26T13:55:56.464 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster1 2023-08-26T13:55:56.464 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T13:55:56.464 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=image_27 2023-08-26T13:55:56.464 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local log=/tmp/tmp.TfDn6BcQ3N/cluster1-mirror-image_27.status 2023-08-26T13:55:56.464 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_27 --format xml 2023-08-26T13:55:56.465 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-08-26T13:55:56.509 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-08-26T13:55:56.511 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local primary_snapshot_id=94 2023-08-26T13:55:56.511 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ test 171 = 94 2023-08-26T13:55:56.511 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-08-26T13:55:56.512 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ sleep 0.4 2023-08-26T13:55:56.912 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_27 /tmp/tmp.TfDn6BcQ3N/cluster1-mirror-image_27.status 2023-08-26T13:55:56.913 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster1 2023-08-26T13:55:56.913 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T13:55:56.913 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=image_27 2023-08-26T13:55:56.913 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local log=/tmp/tmp.TfDn6BcQ3N/cluster1-mirror-image_27.status 2023-08-26T13:55:56.914 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_27 --format xml 2023-08-26T13:55:56.914 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-08-26T13:55:56.960 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ true 2023-08-26T13:55:56.961 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-08-26T13:55:56.962 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:-:1.1: Document is empty 2023-08-26T13:55:56.962 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr: 2023-08-26T13:55:56.962 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:^ 2023-08-26T13:55:56.963 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local primary_snapshot_id= 2023-08-26T13:55:56.963 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ test 171 = '' 2023-08-26T13:55:56.963 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-08-26T13:55:56.964 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ sleep 0.8 2023-08-26T13:55:57.764 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_27 /tmp/tmp.TfDn6BcQ3N/cluster1-mirror-image_27.status 2023-08-26T13:55:57.764 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster1 2023-08-26T13:55:57.764 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T13:55:57.765 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=image_27 2023-08-26T13:55:57.765 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local log=/tmp/tmp.TfDn6BcQ3N/cluster1-mirror-image_27.status 2023-08-26T13:55:57.765 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_27 --format xml 2023-08-26T13:55:57.765 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-08-26T13:55:57.829 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-08-26T13:55:57.832 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local primary_snapshot_id=171 2023-08-26T13:55:57.832 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ test 171 = 171 2023-08-26T13:55:57.832 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ return 0 2023-08-26T13:55:57.832 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ wait_for_snap_present cluster1 mirror image_27 snap 2023-08-26T13:55:57.833 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster1 2023-08-26T13:55:57.833 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T13:55:57.833 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=image_27 2023-08-26T13:55:57.833 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local snap_name=snap 2023-08-26T13:55:57.834 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local s 2023-08-26T13:55:57.834 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2023-08-26T13:55:57.834 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ sleep 1 2023-08-26T13:55:58.832 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rbd --cluster cluster1 info mirror/image_27@snap 2023-08-26T13:55:58.879 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:rbd image 'image_27': 2023-08-26T13:55:58.880 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: size 128 MiB in 32 objects 2023-08-26T13:55:58.880 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: order 22 (4 MiB objects) 2023-08-26T13:55:58.880 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: snapshot_count: 2 2023-08-26T13:55:58.880 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: id: 12b68319ef00 2023-08-26T13:55:58.881 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: block_name_prefix: rbd_data.12b68319ef00 2023-08-26T13:55:58.881 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: format: 2 2023-08-26T13:55:58.881 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: features: layering, exclusive-lock, non-primary 2023-08-26T13:55:58.881 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: op_features: 2023-08-26T13:55:58.881 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: flags: 2023-08-26T13:55:58.882 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: create_timestamp: Sat Aug 26 13:47:03 2023 2023-08-26T13:55:58.882 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: access_timestamp: Sat Aug 26 13:47:03 2023 2023-08-26T13:55:58.882 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: modify_timestamp: Sat Aug 26 13:47:03 2023 2023-08-26T13:55:58.883 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: protected: False 2023-08-26T13:55:58.883 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: mirroring state: enabled 2023-08-26T13:55:58.883 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: mirroring mode: snapshot 2023-08-26T13:55:58.883 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: mirroring global id: bceda8a0-e824-4b77-b33c-1ee70e5c9a49 2023-08-26T13:55:58.884 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: mirroring primary: false 2023-08-26T13:55:58.885 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ return 0 2023-08-26T13:55:58.885 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ compare_image_snaps mirror image_27 snap 2023-08-26T13:55:58.885 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T13:55:58.886 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=image_27 2023-08-26T13:55:58.886 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local snap_name=snap 2023-08-26T13:55:58.886 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local ret=0 2023-08-26T13:55:58.886 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local rmt_export=/tmp/tmp.TfDn6BcQ3N/cluster2-mirror-image_27.export 2023-08-26T13:55:58.886 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local loc_export=/tmp/tmp.TfDn6BcQ3N/cluster1-mirror-image_27.export 2023-08-26T13:55:58.887 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rm -f /tmp/tmp.TfDn6BcQ3N/cluster2-mirror-image_27.export /tmp/tmp.TfDn6BcQ3N/cluster1-mirror-image_27.export 2023-08-26T13:55:58.887 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rbd --cluster cluster2 -p mirror export image_27@snap /tmp/tmp.TfDn6BcQ3N/cluster2-mirror-image_27.export 2023-08-26T13:55:59.068 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-08-26T13:55:59.074 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rbd --cluster cluster1 -p mirror export image_27@snap /tmp/tmp.TfDn6BcQ3N/cluster1-mirror-image_27.export 2023-08-26T13:55:59.311 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-08-26T13:55:59.316 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ cmp /tmp/tmp.TfDn6BcQ3N/cluster2-mirror-image_27.export /tmp/tmp.TfDn6BcQ3N/cluster1-mirror-image_27.export 2023-08-26T13:55:59.370 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rm -f /tmp/tmp.TfDn6BcQ3N/cluster2-mirror-image_27.export /tmp/tmp.TfDn6BcQ3N/cluster1-mirror-image_27.export 2023-08-26T13:55:59.408 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ return 0 2023-08-26T13:55:59.408 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-08-26T13:55:59.408 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ image=image_28 2023-08-26T13:55:59.408 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ create_snap cluster2 mirror image_28 snap 2023-08-26T13:55:59.409 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster2 2023-08-26T13:55:59.409 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T13:55:59.409 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=image_28 2023-08-26T13:55:59.409 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local snap_name=snap 2023-08-26T13:55:59.409 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rbd --cluster cluster2 -p mirror snap create image_28@snap --debug-rbd=20 --debug-journaler=20 2023-08-26T13:56:01.214 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ wait_for_image_replay_started cluster1 mirror image_28 2023-08-26T13:56:01.214 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster1 2023-08-26T13:56:01.214 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T13:56:01.214 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=image_28 2023-08-26T13:56:01.215 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ wait_for_image_replay_state cluster1 mirror image_28 started 2023-08-26T13:56:01.215 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster1 2023-08-26T13:56:01.215 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T13:56:01.215 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=image_28 2023-08-26T13:56:01.215 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local state=started 2023-08-26T13:56:01.216 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local s 2023-08-26T13:56:01.216 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-08-26T13:56:01.216 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ sleep 1 2023-08-26T13:56:02.215 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ test_image_replay_state cluster1 mirror image_28 started 2023-08-26T13:56:02.215 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster1 2023-08-26T13:56:02.215 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T13:56:02.216 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=image_28 2023-08-26T13:56:02.216 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local test_state=started 2023-08-26T13:56:02.216 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local status_result 2023-08-26T13:56:02.216 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local current_state=stopped 2023-08-26T13:56:02.221 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_28 2023-08-26T13:56:02.221 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local cluster_instance=cluster1 2023-08-26T13:56:02.221 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ shift 2023-08-26T13:56:02.221 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ grep -i state 2023-08-26T13:56:02.222 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local cluster=cluster1 2023-08-26T13:56:02.222 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local instance=cluster1 2023-08-26T13:56:02.222 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local loop_instance 2023-08-26T13:56:02.222 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-08-26T13:56:02.222 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ sleep 0 2023-08-26T13:56:02.222 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ '[' cluster1 '!=' cluster1 ']' 2023-08-26T13:56:02.223 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ seq 0 3 2023-08-26T13:56:02.223 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-08-26T13:56:02.223 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_28 2023-08-26T13:56:02.223 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local cluster=cluster1:0 2023-08-26T13:56:02.224 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ shift 2023-08-26T13:56:02.224 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local instance 2023-08-26T13:56:02.224 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ set_cluster_instance cluster1:0 cluster instance 2023-08-26T13:56:02.224 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local val=cluster1:0 2023-08-26T13:56:02.225 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local cluster_var_name=cluster 2023-08-26T13:56:02.225 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local instance_var_name=instance 2023-08-26T13:56:02.225 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ cluster=cluster1 2023-08-26T13:56:02.225 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ instance=0 2023-08-26T13:56:02.225 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ '[' 0 = cluster1:0 ']' 2023-08-26T13:56:02.226 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ eval cluster=cluster1 2023-08-26T13:56:02.226 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ cluster=cluster1 2023-08-26T13:56:02.226 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ eval instance=0 2023-08-26T13:56:02.226 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ instance=0 2023-08-26T13:56:02.226 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ daemon_asok_file cluster1:0 cluster1 2023-08-26T13:56:02.227 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ local local_cluster=cluster1:0 2023-08-26T13:56:02.227 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ local cluster=cluster1 2023-08-26T13:56:02.227 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ local instance 2023-08-26T13:56:02.227 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2023-08-26T13:56:02.227 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ local val=cluster1:0 2023-08-26T13:56:02.228 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ local cluster_var_name=local_cluster 2023-08-26T13:56:02.228 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ local instance_var_name=instance 2023-08-26T13:56:02.228 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ cluster=cluster1 2023-08-26T13:56:02.228 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ instance=0 2023-08-26T13:56:02.228 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ '[' 0 = cluster1:0 ']' 2023-08-26T13:56:02.229 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ eval local_cluster=cluster1 2023-08-26T13:56:02.229 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++++ local_cluster=cluster1 2023-08-26T13:56:02.229 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ eval instance=0 2023-08-26T13:56:02.229 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++++ instance=0 2023-08-26T13:56:02.229 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2023-08-26T13:56:02.242 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-08-26T13:56:02.242 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-08-26T13:56:02.242 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-08-26T13:56:02.243 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_28 2023-08-26T13:56:02.353 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:Invalid command: unused arguments: ['mirror/image_28'] 2023-08-26T13:56:02.353 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:rbd mirror status : get status for rbd mirror 2023-08-26T13:56:02.353 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:admin_socket: invalid command 2023-08-26T13:56:02.356 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-08-26T13:56:02.356 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_28 2023-08-26T13:56:02.357 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local cluster=cluster1:1 2023-08-26T13:56:02.357 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ shift 2023-08-26T13:56:02.357 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local instance 2023-08-26T13:56:02.357 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ set_cluster_instance cluster1:1 cluster instance 2023-08-26T13:56:02.357 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local val=cluster1:1 2023-08-26T13:56:02.358 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local cluster_var_name=cluster 2023-08-26T13:56:02.358 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local instance_var_name=instance 2023-08-26T13:56:02.358 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ cluster=cluster1 2023-08-26T13:56:02.358 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ instance=1 2023-08-26T13:56:02.359 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ '[' 1 = cluster1:1 ']' 2023-08-26T13:56:02.359 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ eval cluster=cluster1 2023-08-26T13:56:02.359 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ cluster=cluster1 2023-08-26T13:56:02.359 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ eval instance=1 2023-08-26T13:56:02.360 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ instance=1 2023-08-26T13:56:02.360 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ daemon_asok_file cluster1:1 cluster1 2023-08-26T13:56:02.360 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ local local_cluster=cluster1:1 2023-08-26T13:56:02.361 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ local cluster=cluster1 2023-08-26T13:56:02.361 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ local instance 2023-08-26T13:56:02.361 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2023-08-26T13:56:02.361 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ local val=cluster1:1 2023-08-26T13:56:02.361 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ local cluster_var_name=local_cluster 2023-08-26T13:56:02.362 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ local instance_var_name=instance 2023-08-26T13:56:02.362 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ cluster=cluster1 2023-08-26T13:56:02.362 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ instance=1 2023-08-26T13:56:02.362 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ '[' 1 = cluster1:1 ']' 2023-08-26T13:56:02.362 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ eval local_cluster=cluster1 2023-08-26T13:56:02.362 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++++ local_cluster=cluster1 2023-08-26T13:56:02.363 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ eval instance=1 2023-08-26T13:56:02.363 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++++ instance=1 2023-08-26T13:56:02.363 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2023-08-26T13:56:02.374 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-08-26T13:56:02.375 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-08-26T13:56:02.375 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-08-26T13:56:02.375 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_28 2023-08-26T13:56:02.489 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ return 0 2023-08-26T13:56:02.490 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ status_result=' "state": "Replaying"' 2023-08-26T13:56:02.490 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ echo ' "state": "Replaying"' 2023-08-26T13:56:02.491 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ grep -i Replaying 2023-08-26T13:56:02.491 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: "state": "Replaying" 2023-08-26T13:56:02.493 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ current_state=started 2023-08-26T13:56:02.493 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ test started = started 2023-08-26T13:56:02.493 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ return 0 2023-08-26T13:56:02.493 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_28 2023-08-26T13:56:02.494 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local local_cluster=cluster1 2023-08-26T13:56:02.494 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster2 2023-08-26T13:56:02.495 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T13:56:02.495 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=image_28 2023-08-26T13:56:02.495 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ '[' snapshot = journal ']' 2023-08-26T13:56:02.495 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ '[' snapshot = snapshot ']' 2023-08-26T13:56:02.495 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_28 2023-08-26T13:56:02.496 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local local_cluster=cluster1 2023-08-26T13:56:02.496 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster2 2023-08-26T13:56:02.496 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T13:56:02.496 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=image_28 2023-08-26T13:56:02.496 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ mkfname cluster2-mirror-image_28.status 2023-08-26T13:56:02.496 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ echo cluster2-mirror-image_28.status 2023-08-26T13:56:02.497 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ sed -e 's|[/ ]|_|g' 2023-08-26T13:56:02.497 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local status_log=/tmp/tmp.TfDn6BcQ3N/cluster2-mirror-image_28.status 2023-08-26T13:56:02.497 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ mkfname cluster1-mirror-image_28.status 2023-08-26T13:56:02.498 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ echo cluster1-mirror-image_28.status 2023-08-26T13:56:02.498 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ sed -e 's|[/ ]|_|g' 2023-08-26T13:56:02.499 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local local_status_log=/tmp/tmp.TfDn6BcQ3N/cluster1-mirror-image_28.status 2023-08-26T13:56:02.499 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ mirror_image_snapshot cluster2 mirror image_28 2023-08-26T13:56:02.499 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster2 2023-08-26T13:56:02.499 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T13:56:02.499 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=image_28 2023-08-26T13:56:02.500 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_28 2023-08-26T13:56:03.223 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:Snapshot ID: 173 2023-08-26T13:56:03.232 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_28 /tmp/tmp.TfDn6BcQ3N/cluster2-mirror-image_28.status 2023-08-26T13:56:03.232 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster2 2023-08-26T13:56:03.232 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T13:56:03.232 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=image_28 2023-08-26T13:56:03.233 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local log=/tmp/tmp.TfDn6BcQ3N/cluster2-mirror-image_28.status 2023-08-26T13:56:03.233 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_28 --format xml 2023-08-26T13:56:03.233 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-08-26T13:56:03.274 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ xmlstarlet sel -t -v //snapshot/id 2023-08-26T13:56:03.276 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local snapshot_id=173 2023-08-26T13:56:03.276 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ true 2023-08-26T13:56:03.276 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-08-26T13:56:03.277 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ sleep 0.2 2023-08-26T13:56:03.477 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_28 /tmp/tmp.TfDn6BcQ3N/cluster1-mirror-image_28.status 2023-08-26T13:56:03.478 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster1 2023-08-26T13:56:03.478 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T13:56:03.478 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=image_28 2023-08-26T13:56:03.478 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local log=/tmp/tmp.TfDn6BcQ3N/cluster1-mirror-image_28.status 2023-08-26T13:56:03.479 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_28 --format xml 2023-08-26T13:56:03.479 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-08-26T13:56:03.529 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-08-26T13:56:03.531 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local primary_snapshot_id=95 2023-08-26T13:56:03.531 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ test 173 = 95 2023-08-26T13:56:03.532 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-08-26T13:56:03.532 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ sleep 0.4 2023-08-26T13:56:03.933 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_28 /tmp/tmp.TfDn6BcQ3N/cluster1-mirror-image_28.status 2023-08-26T13:56:03.933 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster1 2023-08-26T13:56:03.933 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T13:56:03.933 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=image_28 2023-08-26T13:56:03.934 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local log=/tmp/tmp.TfDn6BcQ3N/cluster1-mirror-image_28.status 2023-08-26T13:56:03.934 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_28 --format xml 2023-08-26T13:56:03.934 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-08-26T13:56:03.980 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ true 2023-08-26T13:56:03.980 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-08-26T13:56:03.981 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:-:1.1: Document is empty 2023-08-26T13:56:03.982 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr: 2023-08-26T13:56:03.982 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:^ 2023-08-26T13:56:03.982 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local primary_snapshot_id= 2023-08-26T13:56:03.983 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ test 173 = '' 2023-08-26T13:56:03.983 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-08-26T13:56:03.983 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ sleep 0.8 2023-08-26T13:56:04.586 INFO:tasks.ceph.cluster2.osd.1.smithi124.stderr:2023-08-26T13:56:04.584+0000 7f2a5dd89700 -1 reset not still connected to 0x562662351ee0 2023-08-26T13:56:04.586 INFO:tasks.ceph.cluster2.osd.1.smithi124.stderr:2023-08-26T13:56:04.584+0000 7f2a5dd89700 -1 reset not still connected to 0x5626633b81a0 2023-08-26T13:56:04.586 INFO:tasks.ceph.cluster2.osd.1.smithi124.stderr:2023-08-26T13:56:04.584+0000 7f2a5dd89700 -1 reset not still connected to 0x5626695855f0 2023-08-26T13:56:04.783 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_28 /tmp/tmp.TfDn6BcQ3N/cluster1-mirror-image_28.status 2023-08-26T13:56:04.784 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster1 2023-08-26T13:56:04.784 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T13:56:04.784 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=image_28 2023-08-26T13:56:04.784 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local log=/tmp/tmp.TfDn6BcQ3N/cluster1-mirror-image_28.status 2023-08-26T13:56:04.784 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_28 --format xml 2023-08-26T13:56:04.785 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-08-26T13:56:04.831 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ true 2023-08-26T13:56:04.831 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-08-26T13:56:04.833 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:-:1.1: Document is empty 2023-08-26T13:56:04.833 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr: 2023-08-26T13:56:04.833 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:^ 2023-08-26T13:56:04.834 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local primary_snapshot_id= 2023-08-26T13:56:04.834 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ test 173 = '' 2023-08-26T13:56:04.835 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-08-26T13:56:04.835 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ sleep 1.6 2023-08-26T13:56:06.436 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_28 /tmp/tmp.TfDn6BcQ3N/cluster1-mirror-image_28.status 2023-08-26T13:56:06.436 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster1 2023-08-26T13:56:06.436 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T13:56:06.437 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=image_28 2023-08-26T13:56:06.437 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local log=/tmp/tmp.TfDn6BcQ3N/cluster1-mirror-image_28.status 2023-08-26T13:56:06.437 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_28 --format xml 2023-08-26T13:56:06.437 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-08-26T13:56:06.488 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-08-26T13:56:06.490 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local primary_snapshot_id=173 2023-08-26T13:56:06.491 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ test 173 = 173 2023-08-26T13:56:06.491 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ return 0 2023-08-26T13:56:06.491 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ wait_for_snap_present cluster1 mirror image_28 snap 2023-08-26T13:56:06.491 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster1 2023-08-26T13:56:06.491 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T13:56:06.492 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=image_28 2023-08-26T13:56:06.492 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local snap_name=snap 2023-08-26T13:56:06.492 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local s 2023-08-26T13:56:06.492 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2023-08-26T13:56:06.492 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ sleep 1 2023-08-26T13:56:07.491 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rbd --cluster cluster1 info mirror/image_28@snap 2023-08-26T13:56:07.537 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:rbd image 'image_28': 2023-08-26T13:56:07.537 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: size 128 MiB in 32 objects 2023-08-26T13:56:07.537 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: order 22 (4 MiB objects) 2023-08-26T13:56:07.537 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: snapshot_count: 2 2023-08-26T13:56:07.538 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: id: 12c0428944e3 2023-08-26T13:56:07.538 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: block_name_prefix: rbd_data.12c0428944e3 2023-08-26T13:56:07.538 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: format: 2 2023-08-26T13:56:07.538 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: features: layering, exclusive-lock, non-primary 2023-08-26T13:56:07.538 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: op_features: 2023-08-26T13:56:07.539 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: flags: 2023-08-26T13:56:07.539 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: create_timestamp: Sat Aug 26 13:47:11 2023 2023-08-26T13:56:07.539 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: access_timestamp: Sat Aug 26 13:47:11 2023 2023-08-26T13:56:07.539 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: modify_timestamp: Sat Aug 26 13:47:11 2023 2023-08-26T13:56:07.539 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: protected: False 2023-08-26T13:56:07.540 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: mirroring state: enabled 2023-08-26T13:56:07.540 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: mirroring mode: snapshot 2023-08-26T13:56:07.540 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: mirroring global id: b66f96d3-c7f1-4362-870d-f32a84410a1f 2023-08-26T13:56:07.540 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: mirroring primary: false 2023-08-26T13:56:07.541 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ return 0 2023-08-26T13:56:07.542 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ compare_image_snaps mirror image_28 snap 2023-08-26T13:56:07.542 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T13:56:07.542 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=image_28 2023-08-26T13:56:07.542 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local snap_name=snap 2023-08-26T13:56:07.542 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local ret=0 2023-08-26T13:56:07.543 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local rmt_export=/tmp/tmp.TfDn6BcQ3N/cluster2-mirror-image_28.export 2023-08-26T13:56:07.543 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local loc_export=/tmp/tmp.TfDn6BcQ3N/cluster1-mirror-image_28.export 2023-08-26T13:56:07.543 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rm -f /tmp/tmp.TfDn6BcQ3N/cluster2-mirror-image_28.export /tmp/tmp.TfDn6BcQ3N/cluster1-mirror-image_28.export 2023-08-26T13:56:07.543 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rbd --cluster cluster2 -p mirror export image_28@snap /tmp/tmp.TfDn6BcQ3N/cluster2-mirror-image_28.export 2023-08-26T13:56:07.725 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-08-26T13:56:07.732 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rbd --cluster cluster1 -p mirror export image_28@snap /tmp/tmp.TfDn6BcQ3N/cluster1-mirror-image_28.export 2023-08-26T13:56:07.964 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-08-26T13:56:07.970 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ cmp /tmp/tmp.TfDn6BcQ3N/cluster2-mirror-image_28.export /tmp/tmp.TfDn6BcQ3N/cluster1-mirror-image_28.export 2023-08-26T13:56:08.025 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rm -f /tmp/tmp.TfDn6BcQ3N/cluster2-mirror-image_28.export /tmp/tmp.TfDn6BcQ3N/cluster1-mirror-image_28.export 2023-08-26T13:56:08.062 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ return 0 2023-08-26T13:56:08.062 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-08-26T13:56:08.063 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ image=image_29 2023-08-26T13:56:08.063 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ create_snap cluster2 mirror image_29 snap 2023-08-26T13:56:08.063 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster2 2023-08-26T13:56:08.063 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T13:56:08.064 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=image_29 2023-08-26T13:56:08.064 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local snap_name=snap 2023-08-26T13:56:08.064 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rbd --cluster cluster2 -p mirror snap create image_29@snap --debug-rbd=20 --debug-journaler=20 2023-08-26T13:56:09.232 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ wait_for_image_replay_started cluster1 mirror image_29 2023-08-26T13:56:09.232 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster1 2023-08-26T13:56:09.232 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T13:56:09.232 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=image_29 2023-08-26T13:56:09.232 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ wait_for_image_replay_state cluster1 mirror image_29 started 2023-08-26T13:56:09.233 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster1 2023-08-26T13:56:09.233 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T13:56:09.233 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=image_29 2023-08-26T13:56:09.233 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local state=started 2023-08-26T13:56:09.233 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local s 2023-08-26T13:56:09.234 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-08-26T13:56:09.234 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ sleep 1 2023-08-26T13:56:10.233 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ test_image_replay_state cluster1 mirror image_29 started 2023-08-26T13:56:10.233 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster1 2023-08-26T13:56:10.233 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T13:56:10.233 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=image_29 2023-08-26T13:56:10.234 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local test_state=started 2023-08-26T13:56:10.234 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local status_result 2023-08-26T13:56:10.234 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local current_state=stopped 2023-08-26T13:56:10.235 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_29 2023-08-26T13:56:10.235 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ grep -i state 2023-08-26T13:56:10.236 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local cluster_instance=cluster1 2023-08-26T13:56:10.236 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ shift 2023-08-26T13:56:10.236 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local cluster=cluster1 2023-08-26T13:56:10.236 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local instance=cluster1 2023-08-26T13:56:10.236 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local loop_instance 2023-08-26T13:56:10.237 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-08-26T13:56:10.237 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ sleep 0 2023-08-26T13:56:10.237 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ '[' cluster1 '!=' cluster1 ']' 2023-08-26T13:56:10.237 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ seq 0 3 2023-08-26T13:56:10.239 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-08-26T13:56:10.239 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_29 2023-08-26T13:56:10.239 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local cluster=cluster1:0 2023-08-26T13:56:10.239 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ shift 2023-08-26T13:56:10.240 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local instance 2023-08-26T13:56:10.240 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ set_cluster_instance cluster1:0 cluster instance 2023-08-26T13:56:10.240 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local val=cluster1:0 2023-08-26T13:56:10.240 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local cluster_var_name=cluster 2023-08-26T13:56:10.240 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local instance_var_name=instance 2023-08-26T13:56:10.241 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ cluster=cluster1 2023-08-26T13:56:10.241 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ instance=0 2023-08-26T13:56:10.241 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ '[' 0 = cluster1:0 ']' 2023-08-26T13:56:10.241 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ eval cluster=cluster1 2023-08-26T13:56:10.241 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ cluster=cluster1 2023-08-26T13:56:10.241 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ eval instance=0 2023-08-26T13:56:10.243 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ instance=0 2023-08-26T13:56:10.243 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ daemon_asok_file cluster1:0 cluster1 2023-08-26T13:56:10.244 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ local local_cluster=cluster1:0 2023-08-26T13:56:10.244 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ local cluster=cluster1 2023-08-26T13:56:10.244 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ local instance 2023-08-26T13:56:10.244 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2023-08-26T13:56:10.244 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ local val=cluster1:0 2023-08-26T13:56:10.245 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ local cluster_var_name=local_cluster 2023-08-26T13:56:10.245 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ local instance_var_name=instance 2023-08-26T13:56:10.245 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ cluster=cluster1 2023-08-26T13:56:10.245 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ instance=0 2023-08-26T13:56:10.245 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ '[' 0 = cluster1:0 ']' 2023-08-26T13:56:10.245 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ eval local_cluster=cluster1 2023-08-26T13:56:10.246 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++++ local_cluster=cluster1 2023-08-26T13:56:10.246 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ eval instance=0 2023-08-26T13:56:10.246 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++++ instance=0 2023-08-26T13:56:10.246 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2023-08-26T13:56:10.259 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-08-26T13:56:10.259 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-08-26T13:56:10.259 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-08-26T13:56:10.260 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_29 2023-08-26T13:56:10.373 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:Invalid command: unused arguments: ['mirror/image_29'] 2023-08-26T13:56:10.373 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:rbd mirror status : get status for rbd mirror 2023-08-26T13:56:10.373 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:admin_socket: invalid command 2023-08-26T13:56:10.376 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-08-26T13:56:10.376 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_29 2023-08-26T13:56:10.377 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local cluster=cluster1:1 2023-08-26T13:56:10.377 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ shift 2023-08-26T13:56:10.377 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local instance 2023-08-26T13:56:10.377 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ set_cluster_instance cluster1:1 cluster instance 2023-08-26T13:56:10.377 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local val=cluster1:1 2023-08-26T13:56:10.378 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local cluster_var_name=cluster 2023-08-26T13:56:10.378 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local instance_var_name=instance 2023-08-26T13:56:10.378 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ cluster=cluster1 2023-08-26T13:56:10.378 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ instance=1 2023-08-26T13:56:10.378 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ '[' 1 = cluster1:1 ']' 2023-08-26T13:56:10.379 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ eval cluster=cluster1 2023-08-26T13:56:10.379 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ cluster=cluster1 2023-08-26T13:56:10.379 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ eval instance=1 2023-08-26T13:56:10.379 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ instance=1 2023-08-26T13:56:10.379 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ daemon_asok_file cluster1:1 cluster1 2023-08-26T13:56:10.380 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ local local_cluster=cluster1:1 2023-08-26T13:56:10.380 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ local cluster=cluster1 2023-08-26T13:56:10.380 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ local instance 2023-08-26T13:56:10.380 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2023-08-26T13:56:10.380 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ local val=cluster1:1 2023-08-26T13:56:10.381 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ local cluster_var_name=local_cluster 2023-08-26T13:56:10.381 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ local instance_var_name=instance 2023-08-26T13:56:10.381 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ cluster=cluster1 2023-08-26T13:56:10.381 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ instance=1 2023-08-26T13:56:10.382 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ '[' 1 = cluster1:1 ']' 2023-08-26T13:56:10.382 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ eval local_cluster=cluster1 2023-08-26T13:56:10.382 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++++ local_cluster=cluster1 2023-08-26T13:56:10.382 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ eval instance=1 2023-08-26T13:56:10.382 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++++ instance=1 2023-08-26T13:56:10.382 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2023-08-26T13:56:10.394 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-08-26T13:56:10.395 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-08-26T13:56:10.395 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-08-26T13:56:10.395 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_29 2023-08-26T13:56:10.501 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:Invalid command: unused arguments: ['mirror/image_29'] 2023-08-26T13:56:10.501 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:rbd mirror status : get status for rbd mirror 2023-08-26T13:56:10.501 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:admin_socket: invalid command 2023-08-26T13:56:10.504 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-08-26T13:56:10.504 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/image_29 2023-08-26T13:56:10.505 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local cluster=cluster1:2 2023-08-26T13:56:10.505 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ shift 2023-08-26T13:56:10.505 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local instance 2023-08-26T13:56:10.505 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ set_cluster_instance cluster1:2 cluster instance 2023-08-26T13:56:10.505 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local val=cluster1:2 2023-08-26T13:56:10.506 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local cluster_var_name=cluster 2023-08-26T13:56:10.506 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local instance_var_name=instance 2023-08-26T13:56:10.506 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ cluster=cluster1 2023-08-26T13:56:10.506 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ instance=2 2023-08-26T13:56:10.506 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ '[' 2 = cluster1:2 ']' 2023-08-26T13:56:10.506 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ eval cluster=cluster1 2023-08-26T13:56:10.507 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ cluster=cluster1 2023-08-26T13:56:10.507 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ eval instance=2 2023-08-26T13:56:10.507 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ instance=2 2023-08-26T13:56:10.507 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ daemon_asok_file cluster1:2 cluster1 2023-08-26T13:56:10.507 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ local local_cluster=cluster1:2 2023-08-26T13:56:10.508 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ local cluster=cluster1 2023-08-26T13:56:10.508 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ local instance 2023-08-26T13:56:10.508 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2023-08-26T13:56:10.508 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ local val=cluster1:2 2023-08-26T13:56:10.508 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ local cluster_var_name=local_cluster 2023-08-26T13:56:10.508 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ local instance_var_name=instance 2023-08-26T13:56:10.509 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ cluster=cluster1 2023-08-26T13:56:10.509 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ instance=2 2023-08-26T13:56:10.509 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ '[' 2 = cluster1:2 ']' 2023-08-26T13:56:10.509 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ eval local_cluster=cluster1 2023-08-26T13:56:10.509 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++++ local_cluster=cluster1 2023-08-26T13:56:10.509 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ eval instance=2 2023-08-26T13:56:10.510 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++++ instance=2 2023-08-26T13:56:10.510 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2023-08-26T13:56:10.523 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-08-26T13:56:10.523 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-08-26T13:56:10.523 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-08-26T13:56:10.523 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/image_29 2023-08-26T13:56:10.632 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:Invalid command: unused arguments: ['mirror/image_29'] 2023-08-26T13:56:10.632 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:rbd mirror status : get status for rbd mirror 2023-08-26T13:56:10.632 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:admin_socket: invalid command 2023-08-26T13:56:10.635 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-08-26T13:56:10.636 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ admin_daemon cluster1:3 rbd mirror status mirror/image_29 2023-08-26T13:56:10.636 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local cluster=cluster1:3 2023-08-26T13:56:10.636 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ shift 2023-08-26T13:56:10.636 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local instance 2023-08-26T13:56:10.637 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ set_cluster_instance cluster1:3 cluster instance 2023-08-26T13:56:10.637 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local val=cluster1:3 2023-08-26T13:56:10.637 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local cluster_var_name=cluster 2023-08-26T13:56:10.637 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local instance_var_name=instance 2023-08-26T13:56:10.637 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ cluster=cluster1 2023-08-26T13:56:10.638 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ instance=3 2023-08-26T13:56:10.638 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ '[' 3 = cluster1:3 ']' 2023-08-26T13:56:10.638 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ eval cluster=cluster1 2023-08-26T13:56:10.638 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ cluster=cluster1 2023-08-26T13:56:10.638 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ eval instance=3 2023-08-26T13:56:10.638 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ instance=3 2023-08-26T13:56:10.639 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ daemon_asok_file cluster1:3 cluster1 2023-08-26T13:56:10.639 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ local local_cluster=cluster1:3 2023-08-26T13:56:10.639 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ local cluster=cluster1 2023-08-26T13:56:10.639 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ local instance 2023-08-26T13:56:10.639 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ set_cluster_instance cluster1:3 local_cluster instance 2023-08-26T13:56:10.640 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ local val=cluster1:3 2023-08-26T13:56:10.640 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ local cluster_var_name=local_cluster 2023-08-26T13:56:10.640 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ local instance_var_name=instance 2023-08-26T13:56:10.640 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ cluster=cluster1 2023-08-26T13:56:10.640 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ instance=3 2023-08-26T13:56:10.641 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ '[' 3 = cluster1:3 ']' 2023-08-26T13:56:10.641 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ eval local_cluster=cluster1 2023-08-26T13:56:10.641 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++++ local_cluster=cluster1 2023-08-26T13:56:10.641 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ eval instance=3 2023-08-26T13:56:10.641 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++++ instance=3 2023-08-26T13:56:10.642 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2023-08-26T13:56:10.654 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2023-08-26T13:56:10.654 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2023-08-26T13:56:10.654 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2023-08-26T13:56:10.654 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror status mirror/image_29 2023-08-26T13:56:10.770 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ return 0 2023-08-26T13:56:10.771 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ status_result=' "state": "Replaying"' 2023-08-26T13:56:10.771 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ echo ' "state": "Replaying"' 2023-08-26T13:56:10.771 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ grep -i Replaying 2023-08-26T13:56:10.772 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: "state": "Replaying" 2023-08-26T13:56:10.774 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ current_state=started 2023-08-26T13:56:10.774 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ test started = started 2023-08-26T13:56:10.774 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ return 0 2023-08-26T13:56:10.775 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_29 2023-08-26T13:56:10.775 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local local_cluster=cluster1 2023-08-26T13:56:10.775 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster2 2023-08-26T13:56:10.775 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T13:56:10.776 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=image_29 2023-08-26T13:56:10.776 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ '[' snapshot = journal ']' 2023-08-26T13:56:10.776 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ '[' snapshot = snapshot ']' 2023-08-26T13:56:10.778 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_29 2023-08-26T13:56:10.778 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local local_cluster=cluster1 2023-08-26T13:56:10.779 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster2 2023-08-26T13:56:10.779 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T13:56:10.779 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=image_29 2023-08-26T13:56:10.779 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ mkfname cluster2-mirror-image_29.status 2023-08-26T13:56:10.779 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ echo cluster2-mirror-image_29.status 2023-08-26T13:56:10.780 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ sed -e 's|[/ ]|_|g' 2023-08-26T13:56:10.780 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local status_log=/tmp/tmp.TfDn6BcQ3N/cluster2-mirror-image_29.status 2023-08-26T13:56:10.780 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ mkfname cluster1-mirror-image_29.status 2023-08-26T13:56:10.780 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ echo cluster1-mirror-image_29.status 2023-08-26T13:56:10.780 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ sed -e 's|[/ ]|_|g' 2023-08-26T13:56:10.781 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local local_status_log=/tmp/tmp.TfDn6BcQ3N/cluster1-mirror-image_29.status 2023-08-26T13:56:10.782 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ mirror_image_snapshot cluster2 mirror image_29 2023-08-26T13:56:10.782 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster2 2023-08-26T13:56:10.782 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T13:56:10.782 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=image_29 2023-08-26T13:56:10.782 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_29 2023-08-26T13:56:11.240 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:Snapshot ID: 175 2023-08-26T13:56:11.249 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_29 /tmp/tmp.TfDn6BcQ3N/cluster2-mirror-image_29.status 2023-08-26T13:56:11.249 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster2 2023-08-26T13:56:11.249 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T13:56:11.249 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=image_29 2023-08-26T13:56:11.250 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local log=/tmp/tmp.TfDn6BcQ3N/cluster2-mirror-image_29.status 2023-08-26T13:56:11.250 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_29 --format xml 2023-08-26T13:56:11.250 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-08-26T13:56:11.292 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ xmlstarlet sel -t -v //snapshot/id 2023-08-26T13:56:11.294 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local snapshot_id=175 2023-08-26T13:56:11.294 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ true 2023-08-26T13:56:11.294 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-08-26T13:56:11.294 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ sleep 0.2 2023-08-26T13:56:11.495 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_29 /tmp/tmp.TfDn6BcQ3N/cluster1-mirror-image_29.status 2023-08-26T13:56:11.495 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster1 2023-08-26T13:56:11.496 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T13:56:11.496 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=image_29 2023-08-26T13:56:11.496 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local log=/tmp/tmp.TfDn6BcQ3N/cluster1-mirror-image_29.status 2023-08-26T13:56:11.496 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_29 --format xml 2023-08-26T13:56:11.496 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-08-26T13:56:11.540 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ true 2023-08-26T13:56:11.540 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-08-26T13:56:11.541 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:-:1.1: Document is empty 2023-08-26T13:56:11.541 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr: 2023-08-26T13:56:11.542 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:^ 2023-08-26T13:56:11.542 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local primary_snapshot_id= 2023-08-26T13:56:11.542 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ test 175 = '' 2023-08-26T13:56:11.543 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-08-26T13:56:11.543 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ sleep 0.4 2023-08-26T13:56:11.943 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_29 /tmp/tmp.TfDn6BcQ3N/cluster1-mirror-image_29.status 2023-08-26T13:56:11.943 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster1 2023-08-26T13:56:11.944 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T13:56:11.944 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=image_29 2023-08-26T13:56:11.944 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local log=/tmp/tmp.TfDn6BcQ3N/cluster1-mirror-image_29.status 2023-08-26T13:56:11.944 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_29 --format xml 2023-08-26T13:56:11.944 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-08-26T13:56:11.989 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ true 2023-08-26T13:56:11.989 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-08-26T13:56:11.990 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:-:1.1: Document is empty 2023-08-26T13:56:11.990 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr: 2023-08-26T13:56:11.990 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:^ 2023-08-26T13:56:11.991 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local primary_snapshot_id= 2023-08-26T13:56:11.991 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ test 175 = '' 2023-08-26T13:56:11.991 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-08-26T13:56:11.992 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ sleep 0.8 2023-08-26T13:56:12.792 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_29 /tmp/tmp.TfDn6BcQ3N/cluster1-mirror-image_29.status 2023-08-26T13:56:12.792 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster1 2023-08-26T13:56:12.793 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T13:56:12.793 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=image_29 2023-08-26T13:56:12.793 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local log=/tmp/tmp.TfDn6BcQ3N/cluster1-mirror-image_29.status 2023-08-26T13:56:12.793 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_29 --format xml 2023-08-26T13:56:12.793 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-08-26T13:56:12.838 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-08-26T13:56:12.840 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local primary_snapshot_id=175 2023-08-26T13:56:12.840 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ test 175 = 175 2023-08-26T13:56:12.841 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ return 0 2023-08-26T13:56:12.841 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ wait_for_snap_present cluster1 mirror image_29 snap 2023-08-26T13:56:12.841 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster1 2023-08-26T13:56:12.841 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T13:56:12.841 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=image_29 2023-08-26T13:56:12.842 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local snap_name=snap 2023-08-26T13:56:12.842 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local s 2023-08-26T13:56:12.842 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2023-08-26T13:56:12.842 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ sleep 1 2023-08-26T13:56:13.841 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rbd --cluster cluster1 info mirror/image_29@snap 2023-08-26T13:56:13.884 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:rbd image 'image_29': 2023-08-26T13:56:13.885 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: size 128 MiB in 32 objects 2023-08-26T13:56:13.885 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: order 22 (4 MiB objects) 2023-08-26T13:56:13.885 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: snapshot_count: 2 2023-08-26T13:56:13.886 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: id: 10b7ebd0aabb 2023-08-26T13:56:13.886 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: block_name_prefix: rbd_data.10b7ebd0aabb 2023-08-26T13:56:13.886 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: format: 2 2023-08-26T13:56:13.887 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: features: layering, exclusive-lock, non-primary 2023-08-26T13:56:13.887 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: op_features: 2023-08-26T13:56:13.888 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: flags: 2023-08-26T13:56:13.888 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: create_timestamp: Sat Aug 26 13:47:26 2023 2023-08-26T13:56:13.888 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: access_timestamp: Sat Aug 26 13:47:26 2023 2023-08-26T13:56:13.889 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: modify_timestamp: Sat Aug 26 13:47:26 2023 2023-08-26T13:56:13.889 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: protected: False 2023-08-26T13:56:13.889 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: mirroring state: enabled 2023-08-26T13:56:13.890 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: mirroring mode: snapshot 2023-08-26T13:56:13.890 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: mirroring global id: 2286d3a2-d0dc-442e-9a23-9c83c4633272 2023-08-26T13:56:13.890 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: mirroring primary: false 2023-08-26T13:56:13.892 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ return 0 2023-08-26T13:56:13.893 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ compare_image_snaps mirror image_29 snap 2023-08-26T13:56:13.893 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T13:56:13.893 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=image_29 2023-08-26T13:56:13.894 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local snap_name=snap 2023-08-26T13:56:13.894 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local ret=0 2023-08-26T13:56:13.895 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local rmt_export=/tmp/tmp.TfDn6BcQ3N/cluster2-mirror-image_29.export 2023-08-26T13:56:13.895 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local loc_export=/tmp/tmp.TfDn6BcQ3N/cluster1-mirror-image_29.export 2023-08-26T13:56:13.895 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rm -f /tmp/tmp.TfDn6BcQ3N/cluster2-mirror-image_29.export /tmp/tmp.TfDn6BcQ3N/cluster1-mirror-image_29.export 2023-08-26T13:56:13.896 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rbd --cluster cluster2 -p mirror export image_29@snap /tmp/tmp.TfDn6BcQ3N/cluster2-mirror-image_29.export 2023-08-26T13:56:14.061 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-08-26T13:56:14.066 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rbd --cluster cluster1 -p mirror export image_29@snap /tmp/tmp.TfDn6BcQ3N/cluster1-mirror-image_29.export 2023-08-26T13:56:14.306 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-08-26T13:56:14.313 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ cmp /tmp/tmp.TfDn6BcQ3N/cluster2-mirror-image_29.export /tmp/tmp.TfDn6BcQ3N/cluster1-mirror-image_29.export 2023-08-26T13:56:14.368 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rm -f /tmp/tmp.TfDn6BcQ3N/cluster2-mirror-image_29.export /tmp/tmp.TfDn6BcQ3N/cluster1-mirror-image_29.export 2023-08-26T13:56:14.405 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ return 0 2023-08-26T13:56:14.406 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-08-26T13:56:14.406 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ image=image_30 2023-08-26T13:56:14.406 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ create_snap cluster2 mirror image_30 snap 2023-08-26T13:56:14.406 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster2 2023-08-26T13:56:14.407 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T13:56:14.407 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=image_30 2023-08-26T13:56:14.407 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local snap_name=snap 2023-08-26T13:56:14.407 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rbd --cluster cluster2 -p mirror snap create image_30@snap --debug-rbd=20 --debug-journaler=20 2023-08-26T13:56:16.190 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ wait_for_image_replay_started cluster1 mirror image_30 2023-08-26T13:56:16.190 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster1 2023-08-26T13:56:16.191 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T13:56:16.191 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=image_30 2023-08-26T13:56:16.191 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ wait_for_image_replay_state cluster1 mirror image_30 started 2023-08-26T13:56:16.191 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster1 2023-08-26T13:56:16.191 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T13:56:16.192 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=image_30 2023-08-26T13:56:16.192 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local state=started 2023-08-26T13:56:16.192 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local s 2023-08-26T13:56:16.192 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-08-26T13:56:16.192 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ sleep 1 2023-08-26T13:56:17.191 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ test_image_replay_state cluster1 mirror image_30 started 2023-08-26T13:56:17.192 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster1 2023-08-26T13:56:17.192 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T13:56:17.192 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=image_30 2023-08-26T13:56:17.192 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local test_state=started 2023-08-26T13:56:17.192 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local status_result 2023-08-26T13:56:17.193 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local current_state=stopped 2023-08-26T13:56:17.195 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_30 2023-08-26T13:56:17.195 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local cluster_instance=cluster1 2023-08-26T13:56:17.195 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ grep -i state 2023-08-26T13:56:17.196 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ shift 2023-08-26T13:56:17.196 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local cluster=cluster1 2023-08-26T13:56:17.196 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local instance=cluster1 2023-08-26T13:56:17.196 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local loop_instance 2023-08-26T13:56:17.196 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-08-26T13:56:17.197 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ sleep 0 2023-08-26T13:56:17.197 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ '[' cluster1 '!=' cluster1 ']' 2023-08-26T13:56:17.197 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ seq 0 3 2023-08-26T13:56:17.197 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-08-26T13:56:17.197 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_30 2023-08-26T13:56:17.197 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local cluster=cluster1:0 2023-08-26T13:56:17.198 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ shift 2023-08-26T13:56:17.198 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local instance 2023-08-26T13:56:17.198 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ set_cluster_instance cluster1:0 cluster instance 2023-08-26T13:56:17.198 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local val=cluster1:0 2023-08-26T13:56:17.199 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local cluster_var_name=cluster 2023-08-26T13:56:17.199 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local instance_var_name=instance 2023-08-26T13:56:17.199 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ cluster=cluster1 2023-08-26T13:56:17.199 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ instance=0 2023-08-26T13:56:17.199 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ '[' 0 = cluster1:0 ']' 2023-08-26T13:56:17.199 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ eval cluster=cluster1 2023-08-26T13:56:17.200 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ cluster=cluster1 2023-08-26T13:56:17.200 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ eval instance=0 2023-08-26T13:56:17.200 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ instance=0 2023-08-26T13:56:17.202 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ daemon_asok_file cluster1:0 cluster1 2023-08-26T13:56:17.202 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ local local_cluster=cluster1:0 2023-08-26T13:56:17.202 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ local cluster=cluster1 2023-08-26T13:56:17.202 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ local instance 2023-08-26T13:56:17.202 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2023-08-26T13:56:17.202 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ local val=cluster1:0 2023-08-26T13:56:17.203 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ local cluster_var_name=local_cluster 2023-08-26T13:56:17.203 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ local instance_var_name=instance 2023-08-26T13:56:17.203 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ cluster=cluster1 2023-08-26T13:56:17.203 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ instance=0 2023-08-26T13:56:17.203 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ '[' 0 = cluster1:0 ']' 2023-08-26T13:56:17.204 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ eval local_cluster=cluster1 2023-08-26T13:56:17.204 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++++ local_cluster=cluster1 2023-08-26T13:56:17.204 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ eval instance=0 2023-08-26T13:56:17.204 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++++ instance=0 2023-08-26T13:56:17.204 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2023-08-26T13:56:17.217 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-08-26T13:56:17.218 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-08-26T13:56:17.218 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-08-26T13:56:17.218 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_30 2023-08-26T13:56:17.329 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:Invalid command: unused arguments: ['mirror/image_30'] 2023-08-26T13:56:17.330 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:rbd mirror status : get status for rbd mirror 2023-08-26T13:56:17.330 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:admin_socket: invalid command 2023-08-26T13:56:17.332 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-08-26T13:56:17.333 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_30 2023-08-26T13:56:17.333 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local cluster=cluster1:1 2023-08-26T13:56:17.333 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ shift 2023-08-26T13:56:17.333 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local instance 2023-08-26T13:56:17.333 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ set_cluster_instance cluster1:1 cluster instance 2023-08-26T13:56:17.334 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local val=cluster1:1 2023-08-26T13:56:17.334 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local cluster_var_name=cluster 2023-08-26T13:56:17.334 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local instance_var_name=instance 2023-08-26T13:56:17.334 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ cluster=cluster1 2023-08-26T13:56:17.334 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ instance=1 2023-08-26T13:56:17.335 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ '[' 1 = cluster1:1 ']' 2023-08-26T13:56:17.335 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ eval cluster=cluster1 2023-08-26T13:56:17.335 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ cluster=cluster1 2023-08-26T13:56:17.335 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ eval instance=1 2023-08-26T13:56:17.335 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ instance=1 2023-08-26T13:56:17.336 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ daemon_asok_file cluster1:1 cluster1 2023-08-26T13:56:17.336 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ local local_cluster=cluster1:1 2023-08-26T13:56:17.336 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ local cluster=cluster1 2023-08-26T13:56:17.336 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ local instance 2023-08-26T13:56:17.336 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2023-08-26T13:56:17.337 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ local val=cluster1:1 2023-08-26T13:56:17.337 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ local cluster_var_name=local_cluster 2023-08-26T13:56:17.337 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ local instance_var_name=instance 2023-08-26T13:56:17.337 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ cluster=cluster1 2023-08-26T13:56:17.337 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ instance=1 2023-08-26T13:56:17.337 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ '[' 1 = cluster1:1 ']' 2023-08-26T13:56:17.338 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ eval local_cluster=cluster1 2023-08-26T13:56:17.338 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++++ local_cluster=cluster1 2023-08-26T13:56:17.338 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ eval instance=1 2023-08-26T13:56:17.338 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++++ instance=1 2023-08-26T13:56:17.338 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2023-08-26T13:56:17.352 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-08-26T13:56:17.352 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-08-26T13:56:17.352 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-08-26T13:56:17.353 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_30 2023-08-26T13:56:17.460 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:Invalid command: unused arguments: ['mirror/image_30'] 2023-08-26T13:56:17.460 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:rbd mirror status : get status for rbd mirror 2023-08-26T13:56:17.460 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:admin_socket: invalid command 2023-08-26T13:56:17.463 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-08-26T13:56:17.463 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/image_30 2023-08-26T13:56:17.463 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local cluster=cluster1:2 2023-08-26T13:56:17.463 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ shift 2023-08-26T13:56:17.464 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local instance 2023-08-26T13:56:17.464 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ set_cluster_instance cluster1:2 cluster instance 2023-08-26T13:56:17.464 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local val=cluster1:2 2023-08-26T13:56:17.464 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local cluster_var_name=cluster 2023-08-26T13:56:17.464 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local instance_var_name=instance 2023-08-26T13:56:17.465 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ cluster=cluster1 2023-08-26T13:56:17.465 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ instance=2 2023-08-26T13:56:17.465 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ '[' 2 = cluster1:2 ']' 2023-08-26T13:56:17.465 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ eval cluster=cluster1 2023-08-26T13:56:17.465 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ cluster=cluster1 2023-08-26T13:56:17.465 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ eval instance=2 2023-08-26T13:56:17.466 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ instance=2 2023-08-26T13:56:17.466 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ daemon_asok_file cluster1:2 cluster1 2023-08-26T13:56:17.466 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ local local_cluster=cluster1:2 2023-08-26T13:56:17.466 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ local cluster=cluster1 2023-08-26T13:56:17.466 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ local instance 2023-08-26T13:56:17.467 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2023-08-26T13:56:17.467 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ local val=cluster1:2 2023-08-26T13:56:17.467 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ local cluster_var_name=local_cluster 2023-08-26T13:56:17.467 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ local instance_var_name=instance 2023-08-26T13:56:17.467 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ cluster=cluster1 2023-08-26T13:56:17.467 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ instance=2 2023-08-26T13:56:17.468 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ '[' 2 = cluster1:2 ']' 2023-08-26T13:56:17.468 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ eval local_cluster=cluster1 2023-08-26T13:56:17.468 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++++ local_cluster=cluster1 2023-08-26T13:56:17.468 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ eval instance=2 2023-08-26T13:56:17.468 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++++ instance=2 2023-08-26T13:56:17.469 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2023-08-26T13:56:17.481 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-08-26T13:56:17.481 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-08-26T13:56:17.482 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-08-26T13:56:17.482 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/image_30 2023-08-26T13:56:17.586 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:Invalid command: unused arguments: ['mirror/image_30'] 2023-08-26T13:56:17.586 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:rbd mirror status : get status for rbd mirror 2023-08-26T13:56:17.586 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:admin_socket: invalid command 2023-08-26T13:56:17.589 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-08-26T13:56:17.589 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ admin_daemon cluster1:3 rbd mirror status mirror/image_30 2023-08-26T13:56:17.589 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local cluster=cluster1:3 2023-08-26T13:56:17.589 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ shift 2023-08-26T13:56:17.590 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local instance 2023-08-26T13:56:17.590 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ set_cluster_instance cluster1:3 cluster instance 2023-08-26T13:56:17.590 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local val=cluster1:3 2023-08-26T13:56:17.590 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local cluster_var_name=cluster 2023-08-26T13:56:17.590 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local instance_var_name=instance 2023-08-26T13:56:17.590 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ cluster=cluster1 2023-08-26T13:56:17.591 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ instance=3 2023-08-26T13:56:17.591 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ '[' 3 = cluster1:3 ']' 2023-08-26T13:56:17.591 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ eval cluster=cluster1 2023-08-26T13:56:17.591 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ cluster=cluster1 2023-08-26T13:56:17.591 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ eval instance=3 2023-08-26T13:56:17.592 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ instance=3 2023-08-26T13:56:17.592 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ daemon_asok_file cluster1:3 cluster1 2023-08-26T13:56:17.592 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ local local_cluster=cluster1:3 2023-08-26T13:56:17.592 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ local cluster=cluster1 2023-08-26T13:56:17.592 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ local instance 2023-08-26T13:56:17.592 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ set_cluster_instance cluster1:3 local_cluster instance 2023-08-26T13:56:17.593 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ local val=cluster1:3 2023-08-26T13:56:17.593 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ local cluster_var_name=local_cluster 2023-08-26T13:56:17.593 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ local instance_var_name=instance 2023-08-26T13:56:17.593 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ cluster=cluster1 2023-08-26T13:56:17.593 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ instance=3 2023-08-26T13:56:17.594 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ '[' 3 = cluster1:3 ']' 2023-08-26T13:56:17.594 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ eval local_cluster=cluster1 2023-08-26T13:56:17.594 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++++ local_cluster=cluster1 2023-08-26T13:56:17.594 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ eval instance=3 2023-08-26T13:56:17.595 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++++ instance=3 2023-08-26T13:56:17.595 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2023-08-26T13:56:17.607 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2023-08-26T13:56:17.608 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2023-08-26T13:56:17.608 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2023-08-26T13:56:17.608 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror status mirror/image_30 2023-08-26T13:56:17.727 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ return 0 2023-08-26T13:56:17.728 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ status_result=' "state": "Replaying"' 2023-08-26T13:56:17.728 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ echo ' "state": "Replaying"' 2023-08-26T13:56:17.728 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ grep -i Replaying 2023-08-26T13:56:17.729 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: "state": "Replaying" 2023-08-26T13:56:17.731 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ current_state=started 2023-08-26T13:56:17.731 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ test started = started 2023-08-26T13:56:17.731 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ return 0 2023-08-26T13:56:17.732 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_30 2023-08-26T13:56:17.732 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local local_cluster=cluster1 2023-08-26T13:56:17.732 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster2 2023-08-26T13:56:17.732 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T13:56:17.732 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=image_30 2023-08-26T13:56:17.732 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ '[' snapshot = journal ']' 2023-08-26T13:56:17.733 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ '[' snapshot = snapshot ']' 2023-08-26T13:56:17.733 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_30 2023-08-26T13:56:17.733 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local local_cluster=cluster1 2023-08-26T13:56:17.733 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster2 2023-08-26T13:56:17.733 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T13:56:17.734 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=image_30 2023-08-26T13:56:17.734 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ mkfname cluster2-mirror-image_30.status 2023-08-26T13:56:17.734 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ echo cluster2-mirror-image_30.status 2023-08-26T13:56:17.734 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ sed -e 's|[/ ]|_|g' 2023-08-26T13:56:17.734 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local status_log=/tmp/tmp.TfDn6BcQ3N/cluster2-mirror-image_30.status 2023-08-26T13:56:17.735 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ mkfname cluster1-mirror-image_30.status 2023-08-26T13:56:17.735 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ echo cluster1-mirror-image_30.status 2023-08-26T13:56:17.735 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ sed -e 's|[/ ]|_|g' 2023-08-26T13:56:17.736 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local local_status_log=/tmp/tmp.TfDn6BcQ3N/cluster1-mirror-image_30.status 2023-08-26T13:56:17.736 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ mirror_image_snapshot cluster2 mirror image_30 2023-08-26T13:56:17.737 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster2 2023-08-26T13:56:17.738 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T13:56:17.738 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=image_30 2023-08-26T13:56:17.738 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_30 2023-08-26T13:56:18.192 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:Snapshot ID: 177 2023-08-26T13:56:18.201 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_30 /tmp/tmp.TfDn6BcQ3N/cluster2-mirror-image_30.status 2023-08-26T13:56:18.201 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster2 2023-08-26T13:56:18.201 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T13:56:18.202 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=image_30 2023-08-26T13:56:18.202 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local log=/tmp/tmp.TfDn6BcQ3N/cluster2-mirror-image_30.status 2023-08-26T13:56:18.202 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_30 --format xml 2023-08-26T13:56:18.202 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-08-26T13:56:18.242 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ xmlstarlet sel -t -v //snapshot/id 2023-08-26T13:56:18.244 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local snapshot_id=177 2023-08-26T13:56:18.244 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ true 2023-08-26T13:56:18.245 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-08-26T13:56:18.245 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ sleep 0.2 2023-08-26T13:56:18.446 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_30 /tmp/tmp.TfDn6BcQ3N/cluster1-mirror-image_30.status 2023-08-26T13:56:18.446 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster1 2023-08-26T13:56:18.446 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T13:56:18.446 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=image_30 2023-08-26T13:56:18.446 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local log=/tmp/tmp.TfDn6BcQ3N/cluster1-mirror-image_30.status 2023-08-26T13:56:18.447 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_30 --format xml 2023-08-26T13:56:18.447 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-08-26T13:56:18.491 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ true 2023-08-26T13:56:18.491 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-08-26T13:56:18.492 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:-:1.1: Document is empty 2023-08-26T13:56:18.492 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr: 2023-08-26T13:56:18.492 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:^ 2023-08-26T13:56:18.493 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local primary_snapshot_id= 2023-08-26T13:56:18.493 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ test 177 = '' 2023-08-26T13:56:18.493 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-08-26T13:56:18.494 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ sleep 0.4 2023-08-26T13:56:18.894 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_30 /tmp/tmp.TfDn6BcQ3N/cluster1-mirror-image_30.status 2023-08-26T13:56:18.894 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster1 2023-08-26T13:56:18.895 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T13:56:18.895 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=image_30 2023-08-26T13:56:18.895 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local log=/tmp/tmp.TfDn6BcQ3N/cluster1-mirror-image_30.status 2023-08-26T13:56:18.895 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_30 --format xml 2023-08-26T13:56:18.895 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-08-26T13:56:18.939 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ true 2023-08-26T13:56:18.940 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-08-26T13:56:18.940 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:-:1.1: Document is empty 2023-08-26T13:56:18.941 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr: 2023-08-26T13:56:18.941 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:^ 2023-08-26T13:56:18.941 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local primary_snapshot_id= 2023-08-26T13:56:18.942 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ test 177 = '' 2023-08-26T13:56:18.942 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-08-26T13:56:18.942 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ sleep 0.8 2023-08-26T13:56:19.743 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_30 /tmp/tmp.TfDn6BcQ3N/cluster1-mirror-image_30.status 2023-08-26T13:56:19.743 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster1 2023-08-26T13:56:19.744 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T13:56:19.744 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=image_30 2023-08-26T13:56:19.744 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local log=/tmp/tmp.TfDn6BcQ3N/cluster1-mirror-image_30.status 2023-08-26T13:56:19.745 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_30 --format xml 2023-08-26T13:56:19.745 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-08-26T13:56:19.789 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-08-26T13:56:19.791 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local primary_snapshot_id=177 2023-08-26T13:56:19.792 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ test 177 = 177 2023-08-26T13:56:19.792 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ return 0 2023-08-26T13:56:19.792 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ wait_for_snap_present cluster1 mirror image_30 snap 2023-08-26T13:56:19.793 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster1 2023-08-26T13:56:19.793 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T13:56:19.793 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=image_30 2023-08-26T13:56:19.793 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local snap_name=snap 2023-08-26T13:56:19.794 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local s 2023-08-26T13:56:19.795 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2023-08-26T13:56:19.795 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ sleep 1 2023-08-26T13:56:20.792 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rbd --cluster cluster1 info mirror/image_30@snap 2023-08-26T13:56:20.843 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:rbd image 'image_30': 2023-08-26T13:56:20.843 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: size 128 MiB in 32 objects 2023-08-26T13:56:20.844 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: order 22 (4 MiB objects) 2023-08-26T13:56:20.844 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: snapshot_count: 2 2023-08-26T13:56:20.844 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: id: 12cfd2ae1226 2023-08-26T13:56:20.844 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: block_name_prefix: rbd_data.12cfd2ae1226 2023-08-26T13:56:20.844 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: format: 2 2023-08-26T13:56:20.845 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: features: layering, exclusive-lock, non-primary 2023-08-26T13:56:20.845 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: op_features: 2023-08-26T13:56:20.845 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: flags: 2023-08-26T13:56:20.845 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: create_timestamp: Sat Aug 26 13:47:40 2023 2023-08-26T13:56:20.845 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: access_timestamp: Sat Aug 26 13:47:40 2023 2023-08-26T13:56:20.846 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: modify_timestamp: Sat Aug 26 13:47:40 2023 2023-08-26T13:56:20.846 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: protected: False 2023-08-26T13:56:20.846 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: mirroring state: enabled 2023-08-26T13:56:20.846 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: mirroring mode: snapshot 2023-08-26T13:56:20.846 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: mirroring global id: 530f9c41-4e08-4e1d-99cd-a782ba07a616 2023-08-26T13:56:20.847 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: mirroring primary: false 2023-08-26T13:56:20.848 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ return 0 2023-08-26T13:56:20.848 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ compare_image_snaps mirror image_30 snap 2023-08-26T13:56:20.848 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T13:56:20.848 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=image_30 2023-08-26T13:56:20.849 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local snap_name=snap 2023-08-26T13:56:20.849 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local ret=0 2023-08-26T13:56:20.849 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local rmt_export=/tmp/tmp.TfDn6BcQ3N/cluster2-mirror-image_30.export 2023-08-26T13:56:20.849 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local loc_export=/tmp/tmp.TfDn6BcQ3N/cluster1-mirror-image_30.export 2023-08-26T13:56:20.849 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rm -f /tmp/tmp.TfDn6BcQ3N/cluster2-mirror-image_30.export /tmp/tmp.TfDn6BcQ3N/cluster1-mirror-image_30.export 2023-08-26T13:56:20.849 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rbd --cluster cluster2 -p mirror export image_30@snap /tmp/tmp.TfDn6BcQ3N/cluster2-mirror-image_30.export 2023-08-26T13:56:21.028 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-08-26T13:56:21.034 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rbd --cluster cluster1 -p mirror export image_30@snap /tmp/tmp.TfDn6BcQ3N/cluster1-mirror-image_30.export 2023-08-26T13:56:21.259 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-08-26T13:56:21.265 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ cmp /tmp/tmp.TfDn6BcQ3N/cluster2-mirror-image_30.export /tmp/tmp.TfDn6BcQ3N/cluster1-mirror-image_30.export 2023-08-26T13:56:21.320 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rm -f /tmp/tmp.TfDn6BcQ3N/cluster2-mirror-image_30.export /tmp/tmp.TfDn6BcQ3N/cluster1-mirror-image_30.export 2023-08-26T13:56:21.357 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ return 0 2023-08-26T13:56:21.357 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-08-26T13:56:21.357 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ image=image_31 2023-08-26T13:56:21.357 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ create_snap cluster2 mirror image_31 snap 2023-08-26T13:56:21.358 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster2 2023-08-26T13:56:21.358 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T13:56:21.358 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=image_31 2023-08-26T13:56:21.358 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local snap_name=snap 2023-08-26T13:56:21.359 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rbd --cluster cluster2 -p mirror snap create image_31@snap --debug-rbd=20 --debug-journaler=20 2023-08-26T13:56:23.264 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ wait_for_image_replay_started cluster1 mirror image_31 2023-08-26T13:56:23.264 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster1 2023-08-26T13:56:23.265 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T13:56:23.265 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=image_31 2023-08-26T13:56:23.265 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ wait_for_image_replay_state cluster1 mirror image_31 started 2023-08-26T13:56:23.265 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster1 2023-08-26T13:56:23.266 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T13:56:23.266 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=image_31 2023-08-26T13:56:23.266 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local state=started 2023-08-26T13:56:23.266 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local s 2023-08-26T13:56:23.266 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-08-26T13:56:23.266 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ sleep 1 2023-08-26T13:56:23.529 INFO:tasks.rbd_mirror_thrash:kill cluster2.client.mirror.0 2023-08-26T13:56:23.530 INFO:tasks.rbd_mirror.cluster2.client.mirror.0:Sent signal 15 2023-08-26T13:56:23.530 INFO:tasks.rbd_mirror_thrash:waiting for 1 secs before reviving daemons 2023-08-26T13:56:23.531 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi124.stderr:2023-08-26T13:56:23.530+0000 7f516d537700 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.0 (PID: 124790) UID: 1000 2023-08-26T13:56:24.265 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ test_image_replay_state cluster1 mirror image_31 started 2023-08-26T13:56:24.266 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster1 2023-08-26T13:56:24.266 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T13:56:24.266 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=image_31 2023-08-26T13:56:24.266 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local test_state=started 2023-08-26T13:56:24.266 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local status_result 2023-08-26T13:56:24.267 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local current_state=stopped 2023-08-26T13:56:24.268 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_31 2023-08-26T13:56:24.268 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local cluster_instance=cluster1 2023-08-26T13:56:24.268 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ grep -i state 2023-08-26T13:56:24.268 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ shift 2023-08-26T13:56:24.269 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local cluster=cluster1 2023-08-26T13:56:24.269 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local instance=cluster1 2023-08-26T13:56:24.269 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local loop_instance 2023-08-26T13:56:24.269 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-08-26T13:56:24.269 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ sleep 0 2023-08-26T13:56:24.270 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ '[' cluster1 '!=' cluster1 ']' 2023-08-26T13:56:24.270 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ seq 0 3 2023-08-26T13:56:24.273 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-08-26T13:56:24.273 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_31 2023-08-26T13:56:24.273 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local cluster=cluster1:0 2023-08-26T13:56:24.273 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ shift 2023-08-26T13:56:24.274 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local instance 2023-08-26T13:56:24.274 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ set_cluster_instance cluster1:0 cluster instance 2023-08-26T13:56:24.274 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local val=cluster1:0 2023-08-26T13:56:24.274 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local cluster_var_name=cluster 2023-08-26T13:56:24.274 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local instance_var_name=instance 2023-08-26T13:56:24.274 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ cluster=cluster1 2023-08-26T13:56:24.275 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ instance=0 2023-08-26T13:56:24.275 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ '[' 0 = cluster1:0 ']' 2023-08-26T13:56:24.275 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ eval cluster=cluster1 2023-08-26T13:56:24.275 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ cluster=cluster1 2023-08-26T13:56:24.275 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ eval instance=0 2023-08-26T13:56:24.276 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ instance=0 2023-08-26T13:56:24.276 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ daemon_asok_file cluster1:0 cluster1 2023-08-26T13:56:24.276 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ local local_cluster=cluster1:0 2023-08-26T13:56:24.276 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ local cluster=cluster1 2023-08-26T13:56:24.276 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ local instance 2023-08-26T13:56:24.277 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2023-08-26T13:56:24.277 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ local val=cluster1:0 2023-08-26T13:56:24.277 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ local cluster_var_name=local_cluster 2023-08-26T13:56:24.277 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ local instance_var_name=instance 2023-08-26T13:56:24.277 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ cluster=cluster1 2023-08-26T13:56:24.278 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ instance=0 2023-08-26T13:56:24.278 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ '[' 0 = cluster1:0 ']' 2023-08-26T13:56:24.278 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ eval local_cluster=cluster1 2023-08-26T13:56:24.278 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++++ local_cluster=cluster1 2023-08-26T13:56:24.278 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ eval instance=0 2023-08-26T13:56:24.278 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++++ instance=0 2023-08-26T13:56:24.279 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2023-08-26T13:56:24.292 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-08-26T13:56:24.292 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-08-26T13:56:24.292 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-08-26T13:56:24.293 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_31 2023-08-26T13:56:24.409 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:Invalid command: unused arguments: ['mirror/image_31'] 2023-08-26T13:56:24.409 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:rbd mirror status : get status for rbd mirror 2023-08-26T13:56:24.410 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:admin_socket: invalid command 2023-08-26T13:56:24.412 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-08-26T13:56:24.412 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_31 2023-08-26T13:56:24.413 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local cluster=cluster1:1 2023-08-26T13:56:24.413 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ shift 2023-08-26T13:56:24.413 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local instance 2023-08-26T13:56:24.413 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ set_cluster_instance cluster1:1 cluster instance 2023-08-26T13:56:24.413 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local val=cluster1:1 2023-08-26T13:56:24.414 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local cluster_var_name=cluster 2023-08-26T13:56:24.414 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local instance_var_name=instance 2023-08-26T13:56:24.414 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ cluster=cluster1 2023-08-26T13:56:24.414 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ instance=1 2023-08-26T13:56:24.414 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ '[' 1 = cluster1:1 ']' 2023-08-26T13:56:24.415 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ eval cluster=cluster1 2023-08-26T13:56:24.415 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ cluster=cluster1 2023-08-26T13:56:24.415 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ eval instance=1 2023-08-26T13:56:24.415 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ instance=1 2023-08-26T13:56:24.415 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ daemon_asok_file cluster1:1 cluster1 2023-08-26T13:56:24.416 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ local local_cluster=cluster1:1 2023-08-26T13:56:24.416 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ local cluster=cluster1 2023-08-26T13:56:24.416 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ local instance 2023-08-26T13:56:24.416 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2023-08-26T13:56:24.416 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ local val=cluster1:1 2023-08-26T13:56:24.417 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ local cluster_var_name=local_cluster 2023-08-26T13:56:24.417 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ local instance_var_name=instance 2023-08-26T13:56:24.417 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ cluster=cluster1 2023-08-26T13:56:24.417 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ instance=1 2023-08-26T13:56:24.417 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ '[' 1 = cluster1:1 ']' 2023-08-26T13:56:24.418 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ eval local_cluster=cluster1 2023-08-26T13:56:24.418 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++++ local_cluster=cluster1 2023-08-26T13:56:24.418 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ eval instance=1 2023-08-26T13:56:24.418 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++++ instance=1 2023-08-26T13:56:24.418 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2023-08-26T13:56:24.431 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-08-26T13:56:24.432 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-08-26T13:56:24.432 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-08-26T13:56:24.432 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_31 2023-08-26T13:56:24.531 INFO:tasks.rbd_mirror_thrash:waiting for cluster2.client.mirror.0 2023-08-26T13:56:24.531 INFO:teuthology.orchestra.run:waiting for 600 2023-08-26T13:56:24.532 INFO:tasks.rbd_mirror_thrash:reviving cluster2.client.mirror.0 2023-08-26T13:56:24.532 INFO:tasks.rbd_mirror.cluster2.client.mirror.0:Restarting daemon 2023-08-26T13:56:24.532 DEBUG:teuthology.orchestra.run.smithi124:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.0 2023-08-26T13:56:24.535 INFO:tasks.rbd_mirror.cluster2.client.mirror.0:Started 2023-08-26T13:56:24.535 INFO:tasks.rbd_mirror_thrash:waiting for 69 secs before thrashing 2023-08-26T13:56:24.551 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ return 0 2023-08-26T13:56:24.551 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ status_result=' "state": "Replaying"' 2023-08-26T13:56:24.551 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ echo ' "state": "Replaying"' 2023-08-26T13:56:24.551 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ grep -i Replaying 2023-08-26T13:56:24.552 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: "state": "Replaying" 2023-08-26T13:56:24.554 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ current_state=started 2023-08-26T13:56:24.554 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ test started = started 2023-08-26T13:56:24.555 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ return 0 2023-08-26T13:56:24.555 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_31 2023-08-26T13:56:24.555 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local local_cluster=cluster1 2023-08-26T13:56:24.555 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster2 2023-08-26T13:56:24.555 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T13:56:24.556 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=image_31 2023-08-26T13:56:24.556 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ '[' snapshot = journal ']' 2023-08-26T13:56:24.556 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ '[' snapshot = snapshot ']' 2023-08-26T13:56:24.556 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_31 2023-08-26T13:56:24.557 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local local_cluster=cluster1 2023-08-26T13:56:24.557 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster2 2023-08-26T13:56:24.557 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T13:56:24.557 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=image_31 2023-08-26T13:56:24.558 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ mkfname cluster2-mirror-image_31.status 2023-08-26T13:56:24.558 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ echo cluster2-mirror-image_31.status 2023-08-26T13:56:24.558 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ sed -e 's|[/ ]|_|g' 2023-08-26T13:56:24.558 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local status_log=/tmp/tmp.TfDn6BcQ3N/cluster2-mirror-image_31.status 2023-08-26T13:56:24.559 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ mkfname cluster1-mirror-image_31.status 2023-08-26T13:56:24.559 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ echo cluster1-mirror-image_31.status 2023-08-26T13:56:24.559 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ sed -e 's|[/ ]|_|g' 2023-08-26T13:56:24.560 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local local_status_log=/tmp/tmp.TfDn6BcQ3N/cluster1-mirror-image_31.status 2023-08-26T13:56:24.560 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ mirror_image_snapshot cluster2 mirror image_31 2023-08-26T13:56:24.561 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster2 2023-08-26T13:56:24.561 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T13:56:24.561 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=image_31 2023-08-26T13:56:24.561 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_31 2023-08-26T13:56:25.275 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:Snapshot ID: 179 2023-08-26T13:56:25.284 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_31 /tmp/tmp.TfDn6BcQ3N/cluster2-mirror-image_31.status 2023-08-26T13:56:25.284 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster2 2023-08-26T13:56:25.284 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T13:56:25.285 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=image_31 2023-08-26T13:56:25.285 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local log=/tmp/tmp.TfDn6BcQ3N/cluster2-mirror-image_31.status 2023-08-26T13:56:25.285 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_31 --format xml 2023-08-26T13:56:25.285 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-08-26T13:56:25.327 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ xmlstarlet sel -t -v //snapshot/id 2023-08-26T13:56:25.329 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local snapshot_id=179 2023-08-26T13:56:25.329 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ true 2023-08-26T13:56:25.330 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-08-26T13:56:25.330 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ sleep 0.2 2023-08-26T13:56:25.530 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_31 /tmp/tmp.TfDn6BcQ3N/cluster1-mirror-image_31.status 2023-08-26T13:56:25.531 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster1 2023-08-26T13:56:25.531 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T13:56:25.531 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=image_31 2023-08-26T13:56:25.531 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local log=/tmp/tmp.TfDn6BcQ3N/cluster1-mirror-image_31.status 2023-08-26T13:56:25.532 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_31 --format xml 2023-08-26T13:56:25.532 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-08-26T13:56:25.584 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-08-26T13:56:25.585 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local primary_snapshot_id=98 2023-08-26T13:56:25.586 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ test 179 = 98 2023-08-26T13:56:25.586 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-08-26T13:56:25.586 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ sleep 0.4 2023-08-26T13:56:25.752 INFO:tasks.rbd_mirror_thrash:kill cluster1.client.mirror.0 2023-08-26T13:56:25.753 INFO:tasks.rbd_mirror.cluster1.client.mirror.0:Sent signal 15 2023-08-26T13:56:25.753 INFO:tasks.rbd_mirror_thrash:waiting for 6 secs before reviving daemons 2023-08-26T13:56:25.754 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi124.stderr:2023-08-26T13:56:25.753+0000 7f1da745f700 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.0 (PID: 117959) UID: 1000 2023-08-26T13:56:25.987 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_31 /tmp/tmp.TfDn6BcQ3N/cluster1-mirror-image_31.status 2023-08-26T13:56:25.987 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster1 2023-08-26T13:56:25.988 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T13:56:25.988 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=image_31 2023-08-26T13:56:25.988 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local log=/tmp/tmp.TfDn6BcQ3N/cluster1-mirror-image_31.status 2023-08-26T13:56:25.988 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_31 --format xml 2023-08-26T13:56:25.988 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-08-26T13:56:26.033 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-08-26T13:56:26.034 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local primary_snapshot_id=98 2023-08-26T13:56:26.035 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ test 179 = 98 2023-08-26T13:56:26.035 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-08-26T13:56:26.035 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ sleep 0.8 2023-08-26T13:56:26.836 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_31 /tmp/tmp.TfDn6BcQ3N/cluster1-mirror-image_31.status 2023-08-26T13:56:26.837 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster1 2023-08-26T13:56:26.837 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T13:56:26.837 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=image_31 2023-08-26T13:56:26.837 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local log=/tmp/tmp.TfDn6BcQ3N/cluster1-mirror-image_31.status 2023-08-26T13:56:26.837 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_31 --format xml 2023-08-26T13:56:26.838 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-08-26T13:56:26.886 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ true 2023-08-26T13:56:26.886 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-08-26T13:56:26.888 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:-:1.1: Document is empty 2023-08-26T13:56:26.888 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr: 2023-08-26T13:56:26.888 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:^ 2023-08-26T13:56:26.888 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local primary_snapshot_id= 2023-08-26T13:56:26.889 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ test 179 = '' 2023-08-26T13:56:26.889 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-08-26T13:56:26.889 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ sleep 1.6 2023-08-26T13:56:28.489 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_31 /tmp/tmp.TfDn6BcQ3N/cluster1-mirror-image_31.status 2023-08-26T13:56:28.490 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster1 2023-08-26T13:56:28.490 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T13:56:28.490 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=image_31 2023-08-26T13:56:28.490 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local log=/tmp/tmp.TfDn6BcQ3N/cluster1-mirror-image_31.status 2023-08-26T13:56:28.490 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_31 --format xml 2023-08-26T13:56:28.491 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-08-26T13:56:28.537 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-08-26T13:56:28.539 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local primary_snapshot_id=179 2023-08-26T13:56:28.540 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ test 179 = 179 2023-08-26T13:56:28.540 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ return 0 2023-08-26T13:56:28.540 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ wait_for_snap_present cluster1 mirror image_31 snap 2023-08-26T13:56:28.540 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster1 2023-08-26T13:56:28.540 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T13:56:28.541 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=image_31 2023-08-26T13:56:28.541 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local snap_name=snap 2023-08-26T13:56:28.542 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local s 2023-08-26T13:56:28.542 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2023-08-26T13:56:28.542 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ sleep 1 2023-08-26T13:56:29.540 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rbd --cluster cluster1 info mirror/image_31@snap 2023-08-26T13:56:29.788 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:rbd image 'image_31': 2023-08-26T13:56:29.788 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: size 128 MiB in 32 objects 2023-08-26T13:56:29.789 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: order 22 (4 MiB objects) 2023-08-26T13:56:29.789 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: snapshot_count: 2 2023-08-26T13:56:29.789 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: id: 12c0590a2b16 2023-08-26T13:56:29.789 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: block_name_prefix: rbd_data.12c0590a2b16 2023-08-26T13:56:29.789 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: format: 2 2023-08-26T13:56:29.790 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: features: layering, exclusive-lock, non-primary 2023-08-26T13:56:29.790 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: op_features: 2023-08-26T13:56:29.790 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: flags: 2023-08-26T13:56:29.790 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: create_timestamp: Sat Aug 26 13:47:50 2023 2023-08-26T13:56:29.790 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: access_timestamp: Sat Aug 26 13:47:50 2023 2023-08-26T13:56:29.791 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: modify_timestamp: Sat Aug 26 13:47:50 2023 2023-08-26T13:56:29.791 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: protected: False 2023-08-26T13:56:29.791 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: mirroring state: enabled 2023-08-26T13:56:29.791 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: mirroring mode: snapshot 2023-08-26T13:56:29.791 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: mirroring global id: c5831e57-7926-4f65-bb0c-ef45cd41b0d4 2023-08-26T13:56:29.792 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: mirroring primary: false 2023-08-26T13:56:29.793 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ return 0 2023-08-26T13:56:29.793 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ compare_image_snaps mirror image_31 snap 2023-08-26T13:56:29.793 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T13:56:29.793 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=image_31 2023-08-26T13:56:29.793 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local snap_name=snap 2023-08-26T13:56:29.794 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local ret=0 2023-08-26T13:56:29.794 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local rmt_export=/tmp/tmp.TfDn6BcQ3N/cluster2-mirror-image_31.export 2023-08-26T13:56:29.794 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local loc_export=/tmp/tmp.TfDn6BcQ3N/cluster1-mirror-image_31.export 2023-08-26T13:56:29.794 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rm -f /tmp/tmp.TfDn6BcQ3N/cluster2-mirror-image_31.export /tmp/tmp.TfDn6BcQ3N/cluster1-mirror-image_31.export 2023-08-26T13:56:29.795 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rbd --cluster cluster2 -p mirror export image_31@snap /tmp/tmp.TfDn6BcQ3N/cluster2-mirror-image_31.export 2023-08-26T13:56:29.966 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-08-26T13:56:29.971 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rbd --cluster cluster1 -p mirror export image_31@snap /tmp/tmp.TfDn6BcQ3N/cluster1-mirror-image_31.export 2023-08-26T13:56:30.198 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-08-26T13:56:30.205 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ cmp /tmp/tmp.TfDn6BcQ3N/cluster2-mirror-image_31.export /tmp/tmp.TfDn6BcQ3N/cluster1-mirror-image_31.export 2023-08-26T13:56:30.260 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rm -f /tmp/tmp.TfDn6BcQ3N/cluster2-mirror-image_31.export /tmp/tmp.TfDn6BcQ3N/cluster1-mirror-image_31.export 2023-08-26T13:56:30.298 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ return 0 2023-08-26T13:56:30.298 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-08-26T13:56:30.298 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ image=image_32 2023-08-26T13:56:30.298 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ create_snap cluster2 mirror image_32 snap 2023-08-26T13:56:30.299 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster2 2023-08-26T13:56:30.299 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T13:56:30.299 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=image_32 2023-08-26T13:56:30.299 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local snap_name=snap 2023-08-26T13:56:30.299 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rbd --cluster cluster2 -p mirror snap create image_32@snap --debug-rbd=20 --debug-journaler=20 2023-08-26T13:56:31.754 INFO:tasks.rbd_mirror_thrash:waiting for cluster1.client.mirror.0 2023-08-26T13:56:31.754 INFO:teuthology.orchestra.run:waiting for 600 2023-08-26T13:56:31.755 INFO:tasks.rbd_mirror_thrash:reviving cluster1.client.mirror.0 2023-08-26T13:56:31.755 INFO:tasks.rbd_mirror.cluster1.client.mirror.0:Restarting daemon 2023-08-26T13:56:31.755 DEBUG:teuthology.orchestra.run.smithi124:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.0 2023-08-26T13:56:31.758 INFO:tasks.rbd_mirror.cluster1.client.mirror.0:Started 2023-08-26T13:56:31.758 INFO:tasks.rbd_mirror_thrash:waiting for 75 secs before thrashing 2023-08-26T13:56:32.284 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ wait_for_image_replay_started cluster1 mirror image_32 2023-08-26T13:56:32.284 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster1 2023-08-26T13:56:32.284 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T13:56:32.284 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=image_32 2023-08-26T13:56:32.284 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ wait_for_image_replay_state cluster1 mirror image_32 started 2023-08-26T13:56:32.285 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster1 2023-08-26T13:56:32.285 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T13:56:32.285 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=image_32 2023-08-26T13:56:32.285 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local state=started 2023-08-26T13:56:32.285 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local s 2023-08-26T13:56:32.285 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-08-26T13:56:32.286 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ sleep 1 2023-08-26T13:56:33.285 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ test_image_replay_state cluster1 mirror image_32 started 2023-08-26T13:56:33.285 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster1 2023-08-26T13:56:33.285 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T13:56:33.285 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=image_32 2023-08-26T13:56:33.286 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local test_state=started 2023-08-26T13:56:33.286 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local status_result 2023-08-26T13:56:33.286 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local current_state=stopped 2023-08-26T13:56:33.287 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_32 2023-08-26T13:56:33.288 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ grep -i state 2023-08-26T13:56:33.288 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local cluster_instance=cluster1 2023-08-26T13:56:33.288 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ shift 2023-08-26T13:56:33.288 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local cluster=cluster1 2023-08-26T13:56:33.289 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local instance=cluster1 2023-08-26T13:56:33.289 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local loop_instance 2023-08-26T13:56:33.289 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-08-26T13:56:33.289 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ sleep 0 2023-08-26T13:56:33.289 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ '[' cluster1 '!=' cluster1 ']' 2023-08-26T13:56:33.290 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ seq 0 3 2023-08-26T13:56:33.292 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-08-26T13:56:33.293 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_32 2023-08-26T13:56:33.293 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local cluster=cluster1:0 2023-08-26T13:56:33.293 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ shift 2023-08-26T13:56:33.293 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local instance 2023-08-26T13:56:33.293 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ set_cluster_instance cluster1:0 cluster instance 2023-08-26T13:56:33.294 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local val=cluster1:0 2023-08-26T13:56:33.294 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local cluster_var_name=cluster 2023-08-26T13:56:33.294 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local instance_var_name=instance 2023-08-26T13:56:33.294 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ cluster=cluster1 2023-08-26T13:56:33.294 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ instance=0 2023-08-26T13:56:33.295 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ '[' 0 = cluster1:0 ']' 2023-08-26T13:56:33.295 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ eval cluster=cluster1 2023-08-26T13:56:33.295 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ cluster=cluster1 2023-08-26T13:56:33.295 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ eval instance=0 2023-08-26T13:56:33.295 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ instance=0 2023-08-26T13:56:33.296 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ daemon_asok_file cluster1:0 cluster1 2023-08-26T13:56:33.296 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ local local_cluster=cluster1:0 2023-08-26T13:56:33.296 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ local cluster=cluster1 2023-08-26T13:56:33.296 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ local instance 2023-08-26T13:56:33.296 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2023-08-26T13:56:33.297 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ local val=cluster1:0 2023-08-26T13:56:33.297 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ local cluster_var_name=local_cluster 2023-08-26T13:56:33.297 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ local instance_var_name=instance 2023-08-26T13:56:33.297 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ cluster=cluster1 2023-08-26T13:56:33.297 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ instance=0 2023-08-26T13:56:33.298 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ '[' 0 = cluster1:0 ']' 2023-08-26T13:56:33.298 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ eval local_cluster=cluster1 2023-08-26T13:56:33.298 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++++ local_cluster=cluster1 2023-08-26T13:56:33.298 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ eval instance=0 2023-08-26T13:56:33.298 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++++ instance=0 2023-08-26T13:56:33.299 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2023-08-26T13:56:33.312 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-08-26T13:56:33.312 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-08-26T13:56:33.312 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-08-26T13:56:33.312 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_32 2023-08-26T13:56:33.410 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:Invalid command: unused arguments: ['mirror/image_32'] 2023-08-26T13:56:33.411 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:rbd mirror status : get status for rbd mirror 2023-08-26T13:56:33.411 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:admin_socket: invalid command 2023-08-26T13:56:33.413 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-08-26T13:56:33.414 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_32 2023-08-26T13:56:33.415 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local cluster=cluster1:1 2023-08-26T13:56:33.416 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ shift 2023-08-26T13:56:33.416 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local instance 2023-08-26T13:56:33.416 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ set_cluster_instance cluster1:1 cluster instance 2023-08-26T13:56:33.416 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local val=cluster1:1 2023-08-26T13:56:33.416 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local cluster_var_name=cluster 2023-08-26T13:56:33.417 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local instance_var_name=instance 2023-08-26T13:56:33.417 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ cluster=cluster1 2023-08-26T13:56:33.417 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ instance=1 2023-08-26T13:56:33.417 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ '[' 1 = cluster1:1 ']' 2023-08-26T13:56:33.417 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ eval cluster=cluster1 2023-08-26T13:56:33.418 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ cluster=cluster1 2023-08-26T13:56:33.418 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ eval instance=1 2023-08-26T13:56:33.418 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ instance=1 2023-08-26T13:56:33.418 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ daemon_asok_file cluster1:1 cluster1 2023-08-26T13:56:33.418 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ local local_cluster=cluster1:1 2023-08-26T13:56:33.418 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ local cluster=cluster1 2023-08-26T13:56:33.419 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ local instance 2023-08-26T13:56:33.419 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2023-08-26T13:56:33.419 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ local val=cluster1:1 2023-08-26T13:56:33.419 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ local cluster_var_name=local_cluster 2023-08-26T13:56:33.419 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ local instance_var_name=instance 2023-08-26T13:56:33.420 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ cluster=cluster1 2023-08-26T13:56:33.420 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ instance=1 2023-08-26T13:56:33.420 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ '[' 1 = cluster1:1 ']' 2023-08-26T13:56:33.420 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ eval local_cluster=cluster1 2023-08-26T13:56:33.420 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++++ local_cluster=cluster1 2023-08-26T13:56:33.421 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ eval instance=1 2023-08-26T13:56:33.421 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++++ instance=1 2023-08-26T13:56:33.421 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2023-08-26T13:56:33.432 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-08-26T13:56:33.432 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-08-26T13:56:33.432 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-08-26T13:56:33.433 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_32 2023-08-26T13:56:33.539 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:Invalid command: unused arguments: ['mirror/image_32'] 2023-08-26T13:56:33.539 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:rbd mirror status : get status for rbd mirror 2023-08-26T13:56:33.539 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:admin_socket: invalid command 2023-08-26T13:56:33.542 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-08-26T13:56:33.542 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/image_32 2023-08-26T13:56:33.543 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local cluster=cluster1:2 2023-08-26T13:56:33.543 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ shift 2023-08-26T13:56:33.543 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local instance 2023-08-26T13:56:33.543 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ set_cluster_instance cluster1:2 cluster instance 2023-08-26T13:56:33.543 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local val=cluster1:2 2023-08-26T13:56:33.544 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local cluster_var_name=cluster 2023-08-26T13:56:33.544 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local instance_var_name=instance 2023-08-26T13:56:33.544 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ cluster=cluster1 2023-08-26T13:56:33.544 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ instance=2 2023-08-26T13:56:33.544 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ '[' 2 = cluster1:2 ']' 2023-08-26T13:56:33.545 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ eval cluster=cluster1 2023-08-26T13:56:33.545 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ cluster=cluster1 2023-08-26T13:56:33.545 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ eval instance=2 2023-08-26T13:56:33.545 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ instance=2 2023-08-26T13:56:33.545 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ daemon_asok_file cluster1:2 cluster1 2023-08-26T13:56:33.545 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ local local_cluster=cluster1:2 2023-08-26T13:56:33.546 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ local cluster=cluster1 2023-08-26T13:56:33.546 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ local instance 2023-08-26T13:56:33.546 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2023-08-26T13:56:33.546 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ local val=cluster1:2 2023-08-26T13:56:33.546 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ local cluster_var_name=local_cluster 2023-08-26T13:56:33.547 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ local instance_var_name=instance 2023-08-26T13:56:33.547 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ cluster=cluster1 2023-08-26T13:56:33.547 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ instance=2 2023-08-26T13:56:33.547 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ '[' 2 = cluster1:2 ']' 2023-08-26T13:56:33.547 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ eval local_cluster=cluster1 2023-08-26T13:56:33.548 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++++ local_cluster=cluster1 2023-08-26T13:56:33.548 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ eval instance=2 2023-08-26T13:56:33.548 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++++ instance=2 2023-08-26T13:56:33.548 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2023-08-26T13:56:33.560 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-08-26T13:56:33.561 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-08-26T13:56:33.561 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-08-26T13:56:33.561 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/image_32 2023-08-26T13:56:33.666 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:Invalid command: unused arguments: ['mirror/image_32'] 2023-08-26T13:56:33.666 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:rbd mirror status : get status for rbd mirror 2023-08-26T13:56:33.666 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:admin_socket: invalid command 2023-08-26T13:56:33.669 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-08-26T13:56:33.669 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ admin_daemon cluster1:3 rbd mirror status mirror/image_32 2023-08-26T13:56:33.669 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local cluster=cluster1:3 2023-08-26T13:56:33.670 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ shift 2023-08-26T13:56:33.670 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local instance 2023-08-26T13:56:33.670 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ set_cluster_instance cluster1:3 cluster instance 2023-08-26T13:56:33.670 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local val=cluster1:3 2023-08-26T13:56:33.670 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local cluster_var_name=cluster 2023-08-26T13:56:33.670 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local instance_var_name=instance 2023-08-26T13:56:33.671 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ cluster=cluster1 2023-08-26T13:56:33.671 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ instance=3 2023-08-26T13:56:33.671 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ '[' 3 = cluster1:3 ']' 2023-08-26T13:56:33.672 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ eval cluster=cluster1 2023-08-26T13:56:33.672 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ cluster=cluster1 2023-08-26T13:56:33.672 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ eval instance=3 2023-08-26T13:56:33.672 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ instance=3 2023-08-26T13:56:33.672 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ daemon_asok_file cluster1:3 cluster1 2023-08-26T13:56:33.672 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ local local_cluster=cluster1:3 2023-08-26T13:56:33.673 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ local cluster=cluster1 2023-08-26T13:56:33.673 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ local instance 2023-08-26T13:56:33.673 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ set_cluster_instance cluster1:3 local_cluster instance 2023-08-26T13:56:33.673 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ local val=cluster1:3 2023-08-26T13:56:33.673 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ local cluster_var_name=local_cluster 2023-08-26T13:56:33.674 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ local instance_var_name=instance 2023-08-26T13:56:33.674 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ cluster=cluster1 2023-08-26T13:56:33.674 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ instance=3 2023-08-26T13:56:33.674 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ '[' 3 = cluster1:3 ']' 2023-08-26T13:56:33.674 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ eval local_cluster=cluster1 2023-08-26T13:56:33.675 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++++ local_cluster=cluster1 2023-08-26T13:56:33.675 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ eval instance=3 2023-08-26T13:56:33.675 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++++ instance=3 2023-08-26T13:56:33.675 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2023-08-26T13:56:33.688 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2023-08-26T13:56:33.688 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2023-08-26T13:56:33.688 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2023-08-26T13:56:33.688 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror status mirror/image_32 2023-08-26T13:56:33.792 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:Invalid command: unused arguments: ['mirror/image_32'] 2023-08-26T13:56:33.793 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:rbd mirror status : get status for rbd mirror 2023-08-26T13:56:33.793 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:admin_socket: invalid command 2023-08-26T13:56:33.793 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-08-26T13:56:33.794 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ sleep 1 2023-08-26T13:56:34.795 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ '[' cluster1 '!=' cluster1 ']' 2023-08-26T13:56:34.795 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ seq 0 3 2023-08-26T13:56:34.798 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-08-26T13:56:34.798 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_32 2023-08-26T13:56:34.799 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local cluster=cluster1:0 2023-08-26T13:56:34.799 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ shift 2023-08-26T13:56:34.799 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local instance 2023-08-26T13:56:34.799 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ set_cluster_instance cluster1:0 cluster instance 2023-08-26T13:56:34.799 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local val=cluster1:0 2023-08-26T13:56:34.800 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local cluster_var_name=cluster 2023-08-26T13:56:34.800 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local instance_var_name=instance 2023-08-26T13:56:34.800 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ cluster=cluster1 2023-08-26T13:56:34.800 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ instance=0 2023-08-26T13:56:34.800 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ '[' 0 = cluster1:0 ']' 2023-08-26T13:56:34.800 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ eval cluster=cluster1 2023-08-26T13:56:34.801 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ cluster=cluster1 2023-08-26T13:56:34.801 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ eval instance=0 2023-08-26T13:56:34.801 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ instance=0 2023-08-26T13:56:34.801 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ daemon_asok_file cluster1:0 cluster1 2023-08-26T13:56:34.801 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ local local_cluster=cluster1:0 2023-08-26T13:56:34.802 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ local cluster=cluster1 2023-08-26T13:56:34.802 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ local instance 2023-08-26T13:56:34.802 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2023-08-26T13:56:34.802 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ local val=cluster1:0 2023-08-26T13:56:34.802 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ local cluster_var_name=local_cluster 2023-08-26T13:56:34.802 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ local instance_var_name=instance 2023-08-26T13:56:34.803 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ cluster=cluster1 2023-08-26T13:56:34.803 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ instance=0 2023-08-26T13:56:34.803 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ '[' 0 = cluster1:0 ']' 2023-08-26T13:56:34.803 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ eval local_cluster=cluster1 2023-08-26T13:56:34.803 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++++ local_cluster=cluster1 2023-08-26T13:56:34.804 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ eval instance=0 2023-08-26T13:56:34.804 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++++ instance=0 2023-08-26T13:56:34.804 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2023-08-26T13:56:34.820 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-08-26T13:56:34.820 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-08-26T13:56:34.820 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-08-26T13:56:34.821 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_32 2023-08-26T13:56:34.921 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:Invalid command: unused arguments: ['mirror/image_32'] 2023-08-26T13:56:34.921 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:rbd mirror status : get status for rbd mirror 2023-08-26T13:56:34.921 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:admin_socket: invalid command 2023-08-26T13:56:34.924 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-08-26T13:56:34.924 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_32 2023-08-26T13:56:34.924 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local cluster=cluster1:1 2023-08-26T13:56:34.924 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ shift 2023-08-26T13:56:34.925 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local instance 2023-08-26T13:56:34.925 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ set_cluster_instance cluster1:1 cluster instance 2023-08-26T13:56:34.925 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local val=cluster1:1 2023-08-26T13:56:34.925 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local cluster_var_name=cluster 2023-08-26T13:56:34.925 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local instance_var_name=instance 2023-08-26T13:56:34.926 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ cluster=cluster1 2023-08-26T13:56:34.926 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ instance=1 2023-08-26T13:56:34.926 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ '[' 1 = cluster1:1 ']' 2023-08-26T13:56:34.926 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ eval cluster=cluster1 2023-08-26T13:56:34.926 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ cluster=cluster1 2023-08-26T13:56:34.926 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ eval instance=1 2023-08-26T13:56:34.927 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ instance=1 2023-08-26T13:56:34.927 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ daemon_asok_file cluster1:1 cluster1 2023-08-26T13:56:34.927 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ local local_cluster=cluster1:1 2023-08-26T13:56:34.927 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ local cluster=cluster1 2023-08-26T13:56:34.927 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ local instance 2023-08-26T13:56:34.927 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2023-08-26T13:56:34.928 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ local val=cluster1:1 2023-08-26T13:56:34.928 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ local cluster_var_name=local_cluster 2023-08-26T13:56:34.928 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ local instance_var_name=instance 2023-08-26T13:56:34.928 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ cluster=cluster1 2023-08-26T13:56:34.928 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ instance=1 2023-08-26T13:56:34.929 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ '[' 1 = cluster1:1 ']' 2023-08-26T13:56:34.929 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ eval local_cluster=cluster1 2023-08-26T13:56:34.929 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++++ local_cluster=cluster1 2023-08-26T13:56:34.929 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ eval instance=1 2023-08-26T13:56:34.929 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++++ instance=1 2023-08-26T13:56:34.930 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2023-08-26T13:56:34.943 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-08-26T13:56:34.943 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-08-26T13:56:34.943 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-08-26T13:56:34.943 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_32 2023-08-26T13:56:35.047 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:Invalid command: unused arguments: ['mirror/image_32'] 2023-08-26T13:56:35.048 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:rbd mirror status : get status for rbd mirror 2023-08-26T13:56:35.048 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:admin_socket: invalid command 2023-08-26T13:56:35.050 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-08-26T13:56:35.050 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/image_32 2023-08-26T13:56:35.051 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local cluster=cluster1:2 2023-08-26T13:56:35.051 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ shift 2023-08-26T13:56:35.051 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local instance 2023-08-26T13:56:35.051 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ set_cluster_instance cluster1:2 cluster instance 2023-08-26T13:56:35.052 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local val=cluster1:2 2023-08-26T13:56:35.052 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local cluster_var_name=cluster 2023-08-26T13:56:35.052 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local instance_var_name=instance 2023-08-26T13:56:35.052 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ cluster=cluster1 2023-08-26T13:56:35.052 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ instance=2 2023-08-26T13:56:35.052 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ '[' 2 = cluster1:2 ']' 2023-08-26T13:56:35.053 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ eval cluster=cluster1 2023-08-26T13:56:35.053 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ cluster=cluster1 2023-08-26T13:56:35.053 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ eval instance=2 2023-08-26T13:56:35.053 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ instance=2 2023-08-26T13:56:35.053 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ daemon_asok_file cluster1:2 cluster1 2023-08-26T13:56:35.054 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ local local_cluster=cluster1:2 2023-08-26T13:56:35.054 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ local cluster=cluster1 2023-08-26T13:56:35.054 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ local instance 2023-08-26T13:56:35.054 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2023-08-26T13:56:35.054 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ local val=cluster1:2 2023-08-26T13:56:35.054 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ local cluster_var_name=local_cluster 2023-08-26T13:56:35.055 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ local instance_var_name=instance 2023-08-26T13:56:35.055 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ cluster=cluster1 2023-08-26T13:56:35.055 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ instance=2 2023-08-26T13:56:35.055 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ '[' 2 = cluster1:2 ']' 2023-08-26T13:56:35.055 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ eval local_cluster=cluster1 2023-08-26T13:56:35.056 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++++ local_cluster=cluster1 2023-08-26T13:56:35.056 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ eval instance=2 2023-08-26T13:56:35.056 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++++ instance=2 2023-08-26T13:56:35.056 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2023-08-26T13:56:35.069 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-08-26T13:56:35.069 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-08-26T13:56:35.069 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-08-26T13:56:35.069 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/image_32 2023-08-26T13:56:35.174 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:Invalid command: unused arguments: ['mirror/image_32'] 2023-08-26T13:56:35.174 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:rbd mirror status : get status for rbd mirror 2023-08-26T13:56:35.174 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:admin_socket: invalid command 2023-08-26T13:56:35.177 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-08-26T13:56:35.177 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ admin_daemon cluster1:3 rbd mirror status mirror/image_32 2023-08-26T13:56:35.178 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local cluster=cluster1:3 2023-08-26T13:56:35.178 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ shift 2023-08-26T13:56:35.178 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local instance 2023-08-26T13:56:35.178 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ set_cluster_instance cluster1:3 cluster instance 2023-08-26T13:56:35.178 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local val=cluster1:3 2023-08-26T13:56:35.179 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local cluster_var_name=cluster 2023-08-26T13:56:35.179 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local instance_var_name=instance 2023-08-26T13:56:35.179 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ cluster=cluster1 2023-08-26T13:56:35.179 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ instance=3 2023-08-26T13:56:35.179 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ '[' 3 = cluster1:3 ']' 2023-08-26T13:56:35.179 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ eval cluster=cluster1 2023-08-26T13:56:35.180 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ cluster=cluster1 2023-08-26T13:56:35.180 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ eval instance=3 2023-08-26T13:56:35.180 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ instance=3 2023-08-26T13:56:35.180 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ daemon_asok_file cluster1:3 cluster1 2023-08-26T13:56:35.180 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ local local_cluster=cluster1:3 2023-08-26T13:56:35.181 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ local cluster=cluster1 2023-08-26T13:56:35.181 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ local instance 2023-08-26T13:56:35.181 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ set_cluster_instance cluster1:3 local_cluster instance 2023-08-26T13:56:35.181 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ local val=cluster1:3 2023-08-26T13:56:35.181 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ local cluster_var_name=local_cluster 2023-08-26T13:56:35.181 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ local instance_var_name=instance 2023-08-26T13:56:35.182 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ cluster=cluster1 2023-08-26T13:56:35.182 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ instance=3 2023-08-26T13:56:35.182 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ '[' 3 = cluster1:3 ']' 2023-08-26T13:56:35.182 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ eval local_cluster=cluster1 2023-08-26T13:56:35.182 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++++ local_cluster=cluster1 2023-08-26T13:56:35.182 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ eval instance=3 2023-08-26T13:56:35.182 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++++ instance=3 2023-08-26T13:56:35.183 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2023-08-26T13:56:35.196 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2023-08-26T13:56:35.196 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2023-08-26T13:56:35.196 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2023-08-26T13:56:35.196 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror status mirror/image_32 2023-08-26T13:56:35.303 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:Invalid command: unused arguments: ['mirror/image_32'] 2023-08-26T13:56:35.303 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:rbd mirror status : get status for rbd mirror 2023-08-26T13:56:35.303 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:admin_socket: invalid command 2023-08-26T13:56:35.304 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-08-26T13:56:35.304 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ sleep 2 2023-08-26T13:56:37.305 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ '[' cluster1 '!=' cluster1 ']' 2023-08-26T13:56:37.305 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ seq 0 3 2023-08-26T13:56:37.308 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-08-26T13:56:37.309 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_32 2023-08-26T13:56:37.309 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local cluster=cluster1:0 2023-08-26T13:56:37.309 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ shift 2023-08-26T13:56:37.309 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local instance 2023-08-26T13:56:37.309 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ set_cluster_instance cluster1:0 cluster instance 2023-08-26T13:56:37.310 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local val=cluster1:0 2023-08-26T13:56:37.310 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local cluster_var_name=cluster 2023-08-26T13:56:37.310 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local instance_var_name=instance 2023-08-26T13:56:37.310 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ cluster=cluster1 2023-08-26T13:56:37.310 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ instance=0 2023-08-26T13:56:37.311 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ '[' 0 = cluster1:0 ']' 2023-08-26T13:56:37.311 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ eval cluster=cluster1 2023-08-26T13:56:37.311 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ cluster=cluster1 2023-08-26T13:56:37.311 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ eval instance=0 2023-08-26T13:56:37.311 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ instance=0 2023-08-26T13:56:37.312 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ daemon_asok_file cluster1:0 cluster1 2023-08-26T13:56:37.312 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ local local_cluster=cluster1:0 2023-08-26T13:56:37.312 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ local cluster=cluster1 2023-08-26T13:56:37.312 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ local instance 2023-08-26T13:56:37.312 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2023-08-26T13:56:37.313 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ local val=cluster1:0 2023-08-26T13:56:37.313 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ local cluster_var_name=local_cluster 2023-08-26T13:56:37.313 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ local instance_var_name=instance 2023-08-26T13:56:37.313 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ cluster=cluster1 2023-08-26T13:56:37.314 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ instance=0 2023-08-26T13:56:37.314 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ '[' 0 = cluster1:0 ']' 2023-08-26T13:56:37.314 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ eval local_cluster=cluster1 2023-08-26T13:56:37.314 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++++ local_cluster=cluster1 2023-08-26T13:56:37.314 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ eval instance=0 2023-08-26T13:56:37.315 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++++ instance=0 2023-08-26T13:56:37.315 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2023-08-26T13:56:37.329 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-08-26T13:56:37.330 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-08-26T13:56:37.330 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-08-26T13:56:37.330 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_32 2023-08-26T13:56:37.433 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:Invalid command: unused arguments: ['mirror/image_32'] 2023-08-26T13:56:37.433 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:rbd mirror status : get status for rbd mirror 2023-08-26T13:56:37.433 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:admin_socket: invalid command 2023-08-26T13:56:37.436 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-08-26T13:56:37.436 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_32 2023-08-26T13:56:37.436 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local cluster=cluster1:1 2023-08-26T13:56:37.437 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ shift 2023-08-26T13:56:37.437 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local instance 2023-08-26T13:56:37.437 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ set_cluster_instance cluster1:1 cluster instance 2023-08-26T13:56:37.437 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local val=cluster1:1 2023-08-26T13:56:37.437 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local cluster_var_name=cluster 2023-08-26T13:56:37.438 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local instance_var_name=instance 2023-08-26T13:56:37.438 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ cluster=cluster1 2023-08-26T13:56:37.438 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ instance=1 2023-08-26T13:56:37.438 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ '[' 1 = cluster1:1 ']' 2023-08-26T13:56:37.438 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ eval cluster=cluster1 2023-08-26T13:56:37.438 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ cluster=cluster1 2023-08-26T13:56:37.439 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ eval instance=1 2023-08-26T13:56:37.439 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ instance=1 2023-08-26T13:56:37.439 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ daemon_asok_file cluster1:1 cluster1 2023-08-26T13:56:37.439 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ local local_cluster=cluster1:1 2023-08-26T13:56:37.439 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ local cluster=cluster1 2023-08-26T13:56:37.440 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ local instance 2023-08-26T13:56:37.440 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2023-08-26T13:56:37.440 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ local val=cluster1:1 2023-08-26T13:56:37.440 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ local cluster_var_name=local_cluster 2023-08-26T13:56:37.440 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ local instance_var_name=instance 2023-08-26T13:56:37.440 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ cluster=cluster1 2023-08-26T13:56:37.441 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ instance=1 2023-08-26T13:56:37.441 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ '[' 1 = cluster1:1 ']' 2023-08-26T13:56:37.441 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ eval local_cluster=cluster1 2023-08-26T13:56:37.441 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++++ local_cluster=cluster1 2023-08-26T13:56:37.441 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ eval instance=1 2023-08-26T13:56:37.441 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++++ instance=1 2023-08-26T13:56:37.442 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2023-08-26T13:56:37.455 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-08-26T13:56:37.455 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-08-26T13:56:37.455 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-08-26T13:56:37.455 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_32 2023-08-26T13:56:37.559 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:Invalid command: unused arguments: ['mirror/image_32'] 2023-08-26T13:56:37.559 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:rbd mirror status : get status for rbd mirror 2023-08-26T13:56:37.559 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:admin_socket: invalid command 2023-08-26T13:56:37.562 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-08-26T13:56:37.563 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/image_32 2023-08-26T13:56:37.563 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local cluster=cluster1:2 2023-08-26T13:56:37.563 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ shift 2023-08-26T13:56:37.563 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local instance 2023-08-26T13:56:37.564 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ set_cluster_instance cluster1:2 cluster instance 2023-08-26T13:56:37.564 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local val=cluster1:2 2023-08-26T13:56:37.564 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local cluster_var_name=cluster 2023-08-26T13:56:37.564 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local instance_var_name=instance 2023-08-26T13:56:37.565 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ cluster=cluster1 2023-08-26T13:56:37.565 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ instance=2 2023-08-26T13:56:37.565 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ '[' 2 = cluster1:2 ']' 2023-08-26T13:56:37.565 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ eval cluster=cluster1 2023-08-26T13:56:37.565 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ cluster=cluster1 2023-08-26T13:56:37.566 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ eval instance=2 2023-08-26T13:56:37.566 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ instance=2 2023-08-26T13:56:37.566 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ daemon_asok_file cluster1:2 cluster1 2023-08-26T13:56:37.566 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ local local_cluster=cluster1:2 2023-08-26T13:56:37.567 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ local cluster=cluster1 2023-08-26T13:56:37.567 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ local instance 2023-08-26T13:56:37.567 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2023-08-26T13:56:37.567 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ local val=cluster1:2 2023-08-26T13:56:37.567 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ local cluster_var_name=local_cluster 2023-08-26T13:56:37.568 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ local instance_var_name=instance 2023-08-26T13:56:37.568 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ cluster=cluster1 2023-08-26T13:56:37.568 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ instance=2 2023-08-26T13:56:37.568 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ '[' 2 = cluster1:2 ']' 2023-08-26T13:56:37.568 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ eval local_cluster=cluster1 2023-08-26T13:56:37.569 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++++ local_cluster=cluster1 2023-08-26T13:56:37.569 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ eval instance=2 2023-08-26T13:56:37.569 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++++ instance=2 2023-08-26T13:56:37.569 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2023-08-26T13:56:37.580 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-08-26T13:56:37.581 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-08-26T13:56:37.581 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-08-26T13:56:37.581 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/image_32 2023-08-26T13:56:37.687 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:Invalid command: unused arguments: ['mirror/image_32'] 2023-08-26T13:56:37.687 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:rbd mirror status : get status for rbd mirror 2023-08-26T13:56:37.687 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:admin_socket: invalid command 2023-08-26T13:56:37.691 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-08-26T13:56:37.691 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ admin_daemon cluster1:3 rbd mirror status mirror/image_32 2023-08-26T13:56:37.691 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local cluster=cluster1:3 2023-08-26T13:56:37.691 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ shift 2023-08-26T13:56:37.692 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local instance 2023-08-26T13:56:37.692 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ set_cluster_instance cluster1:3 cluster instance 2023-08-26T13:56:37.692 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local val=cluster1:3 2023-08-26T13:56:37.692 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local cluster_var_name=cluster 2023-08-26T13:56:37.692 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local instance_var_name=instance 2023-08-26T13:56:37.693 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ cluster=cluster1 2023-08-26T13:56:37.693 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ instance=3 2023-08-26T13:56:37.693 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ '[' 3 = cluster1:3 ']' 2023-08-26T13:56:37.693 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ eval cluster=cluster1 2023-08-26T13:56:37.693 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ cluster=cluster1 2023-08-26T13:56:37.694 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ eval instance=3 2023-08-26T13:56:37.694 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ instance=3 2023-08-26T13:56:37.694 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ daemon_asok_file cluster1:3 cluster1 2023-08-26T13:56:37.694 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ local local_cluster=cluster1:3 2023-08-26T13:56:37.694 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ local cluster=cluster1 2023-08-26T13:56:37.694 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ local instance 2023-08-26T13:56:37.695 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ set_cluster_instance cluster1:3 local_cluster instance 2023-08-26T13:56:37.695 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ local val=cluster1:3 2023-08-26T13:56:37.695 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ local cluster_var_name=local_cluster 2023-08-26T13:56:37.695 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ local instance_var_name=instance 2023-08-26T13:56:37.696 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ cluster=cluster1 2023-08-26T13:56:37.696 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ instance=3 2023-08-26T13:56:37.696 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ '[' 3 = cluster1:3 ']' 2023-08-26T13:56:37.696 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ eval local_cluster=cluster1 2023-08-26T13:56:37.696 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++++ local_cluster=cluster1 2023-08-26T13:56:37.697 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ eval instance=3 2023-08-26T13:56:37.697 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++++ instance=3 2023-08-26T13:56:37.697 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2023-08-26T13:56:37.709 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2023-08-26T13:56:37.709 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2023-08-26T13:56:37.709 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2023-08-26T13:56:37.709 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror status mirror/image_32 2023-08-26T13:56:37.815 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:Invalid command: unused arguments: ['mirror/image_32'] 2023-08-26T13:56:37.815 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:rbd mirror status : get status for rbd mirror 2023-08-26T13:56:37.815 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:admin_socket: invalid command 2023-08-26T13:56:37.816 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-08-26T13:56:37.816 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ sleep 4 2023-08-26T13:56:41.817 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ '[' cluster1 '!=' cluster1 ']' 2023-08-26T13:56:41.818 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ seq 0 3 2023-08-26T13:56:41.820 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-08-26T13:56:41.821 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_32 2023-08-26T13:56:41.821 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local cluster=cluster1:0 2023-08-26T13:56:41.821 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ shift 2023-08-26T13:56:41.821 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local instance 2023-08-26T13:56:41.821 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ set_cluster_instance cluster1:0 cluster instance 2023-08-26T13:56:41.821 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local val=cluster1:0 2023-08-26T13:56:41.822 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local cluster_var_name=cluster 2023-08-26T13:56:41.822 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local instance_var_name=instance 2023-08-26T13:56:41.822 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ cluster=cluster1 2023-08-26T13:56:41.822 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ instance=0 2023-08-26T13:56:41.822 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ '[' 0 = cluster1:0 ']' 2023-08-26T13:56:41.823 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ eval cluster=cluster1 2023-08-26T13:56:41.823 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ cluster=cluster1 2023-08-26T13:56:41.823 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ eval instance=0 2023-08-26T13:56:41.823 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ instance=0 2023-08-26T13:56:41.823 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ daemon_asok_file cluster1:0 cluster1 2023-08-26T13:56:41.824 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ local local_cluster=cluster1:0 2023-08-26T13:56:41.824 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ local cluster=cluster1 2023-08-26T13:56:41.824 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ local instance 2023-08-26T13:56:41.824 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2023-08-26T13:56:41.825 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ local val=cluster1:0 2023-08-26T13:56:41.825 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ local cluster_var_name=local_cluster 2023-08-26T13:56:41.825 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ local instance_var_name=instance 2023-08-26T13:56:41.825 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ cluster=cluster1 2023-08-26T13:56:41.826 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ instance=0 2023-08-26T13:56:41.826 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ '[' 0 = cluster1:0 ']' 2023-08-26T13:56:41.826 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ eval local_cluster=cluster1 2023-08-26T13:56:41.826 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++++ local_cluster=cluster1 2023-08-26T13:56:41.826 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ eval instance=0 2023-08-26T13:56:41.827 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++++ instance=0 2023-08-26T13:56:41.827 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2023-08-26T13:56:41.842 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-08-26T13:56:41.842 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-08-26T13:56:41.842 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-08-26T13:56:41.842 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_32 2023-08-26T13:56:41.946 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:Invalid command: unused arguments: ['mirror/image_32'] 2023-08-26T13:56:41.947 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:rbd mirror status : get status for rbd mirror 2023-08-26T13:56:41.947 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:admin_socket: invalid command 2023-08-26T13:56:41.950 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-08-26T13:56:41.950 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_32 2023-08-26T13:56:41.951 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local cluster=cluster1:1 2023-08-26T13:56:41.951 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ shift 2023-08-26T13:56:41.951 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local instance 2023-08-26T13:56:41.951 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ set_cluster_instance cluster1:1 cluster instance 2023-08-26T13:56:41.952 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local val=cluster1:1 2023-08-26T13:56:41.952 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local cluster_var_name=cluster 2023-08-26T13:56:41.952 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local instance_var_name=instance 2023-08-26T13:56:41.952 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ cluster=cluster1 2023-08-26T13:56:41.952 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ instance=1 2023-08-26T13:56:41.953 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ '[' 1 = cluster1:1 ']' 2023-08-26T13:56:41.953 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ eval cluster=cluster1 2023-08-26T13:56:41.953 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ cluster=cluster1 2023-08-26T13:56:41.953 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ eval instance=1 2023-08-26T13:56:41.954 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ instance=1 2023-08-26T13:56:41.954 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ daemon_asok_file cluster1:1 cluster1 2023-08-26T13:56:41.954 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ local local_cluster=cluster1:1 2023-08-26T13:56:41.954 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ local cluster=cluster1 2023-08-26T13:56:41.954 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ local instance 2023-08-26T13:56:41.955 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2023-08-26T13:56:41.955 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ local val=cluster1:1 2023-08-26T13:56:41.955 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ local cluster_var_name=local_cluster 2023-08-26T13:56:41.955 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ local instance_var_name=instance 2023-08-26T13:56:41.955 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ cluster=cluster1 2023-08-26T13:56:41.956 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ instance=1 2023-08-26T13:56:41.956 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ '[' 1 = cluster1:1 ']' 2023-08-26T13:56:41.956 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ eval local_cluster=cluster1 2023-08-26T13:56:41.956 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++++ local_cluster=cluster1 2023-08-26T13:56:41.956 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ eval instance=1 2023-08-26T13:56:41.957 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++++ instance=1 2023-08-26T13:56:41.957 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2023-08-26T13:56:41.968 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-08-26T13:56:41.969 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-08-26T13:56:41.969 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-08-26T13:56:41.969 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_32 2023-08-26T13:56:42.071 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:Invalid command: unused arguments: ['mirror/image_32'] 2023-08-26T13:56:42.071 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:rbd mirror status : get status for rbd mirror 2023-08-26T13:56:42.071 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:admin_socket: invalid command 2023-08-26T13:56:42.074 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-08-26T13:56:42.074 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/image_32 2023-08-26T13:56:42.074 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local cluster=cluster1:2 2023-08-26T13:56:42.075 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ shift 2023-08-26T13:56:42.075 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local instance 2023-08-26T13:56:42.075 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ set_cluster_instance cluster1:2 cluster instance 2023-08-26T13:56:42.075 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local val=cluster1:2 2023-08-26T13:56:42.075 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local cluster_var_name=cluster 2023-08-26T13:56:42.076 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local instance_var_name=instance 2023-08-26T13:56:42.076 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ cluster=cluster1 2023-08-26T13:56:42.076 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ instance=2 2023-08-26T13:56:42.076 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ '[' 2 = cluster1:2 ']' 2023-08-26T13:56:42.076 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ eval cluster=cluster1 2023-08-26T13:56:42.076 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ cluster=cluster1 2023-08-26T13:56:42.077 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ eval instance=2 2023-08-26T13:56:42.077 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ instance=2 2023-08-26T13:56:42.077 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ daemon_asok_file cluster1:2 cluster1 2023-08-26T13:56:42.077 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ local local_cluster=cluster1:2 2023-08-26T13:56:42.077 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ local cluster=cluster1 2023-08-26T13:56:42.078 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ local instance 2023-08-26T13:56:42.078 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2023-08-26T13:56:42.078 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ local val=cluster1:2 2023-08-26T13:56:42.078 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ local cluster_var_name=local_cluster 2023-08-26T13:56:42.078 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ local instance_var_name=instance 2023-08-26T13:56:42.078 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ cluster=cluster1 2023-08-26T13:56:42.079 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ instance=2 2023-08-26T13:56:42.079 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ '[' 2 = cluster1:2 ']' 2023-08-26T13:56:42.079 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ eval local_cluster=cluster1 2023-08-26T13:56:42.079 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++++ local_cluster=cluster1 2023-08-26T13:56:42.079 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ eval instance=2 2023-08-26T13:56:42.080 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++++ instance=2 2023-08-26T13:56:42.080 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2023-08-26T13:56:42.092 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-08-26T13:56:42.092 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-08-26T13:56:42.093 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-08-26T13:56:42.093 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/image_32 2023-08-26T13:56:42.197 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:Invalid command: unused arguments: ['mirror/image_32'] 2023-08-26T13:56:42.198 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:rbd mirror status : get status for rbd mirror 2023-08-26T13:56:42.198 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:admin_socket: invalid command 2023-08-26T13:56:42.202 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-08-26T13:56:42.202 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ admin_daemon cluster1:3 rbd mirror status mirror/image_32 2023-08-26T13:56:42.202 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local cluster=cluster1:3 2023-08-26T13:56:42.203 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ shift 2023-08-26T13:56:42.203 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local instance 2023-08-26T13:56:42.203 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ set_cluster_instance cluster1:3 cluster instance 2023-08-26T13:56:42.203 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local val=cluster1:3 2023-08-26T13:56:42.203 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local cluster_var_name=cluster 2023-08-26T13:56:42.204 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local instance_var_name=instance 2023-08-26T13:56:42.204 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ cluster=cluster1 2023-08-26T13:56:42.204 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ instance=3 2023-08-26T13:56:42.204 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ '[' 3 = cluster1:3 ']' 2023-08-26T13:56:42.204 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ eval cluster=cluster1 2023-08-26T13:56:42.205 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ cluster=cluster1 2023-08-26T13:56:42.205 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ eval instance=3 2023-08-26T13:56:42.205 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ instance=3 2023-08-26T13:56:42.205 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ daemon_asok_file cluster1:3 cluster1 2023-08-26T13:56:42.205 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ local local_cluster=cluster1:3 2023-08-26T13:56:42.206 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ local cluster=cluster1 2023-08-26T13:56:42.206 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ local instance 2023-08-26T13:56:42.206 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ set_cluster_instance cluster1:3 local_cluster instance 2023-08-26T13:56:42.206 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ local val=cluster1:3 2023-08-26T13:56:42.206 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ local cluster_var_name=local_cluster 2023-08-26T13:56:42.207 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ local instance_var_name=instance 2023-08-26T13:56:42.207 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ cluster=cluster1 2023-08-26T13:56:42.207 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ instance=3 2023-08-26T13:56:42.207 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ '[' 3 = cluster1:3 ']' 2023-08-26T13:56:42.207 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ eval local_cluster=cluster1 2023-08-26T13:56:42.208 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++++ local_cluster=cluster1 2023-08-26T13:56:42.208 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ eval instance=3 2023-08-26T13:56:42.208 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++++ instance=3 2023-08-26T13:56:42.208 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2023-08-26T13:56:42.219 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2023-08-26T13:56:42.219 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2023-08-26T13:56:42.220 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2023-08-26T13:56:42.220 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror status mirror/image_32 2023-08-26T13:56:42.325 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:Invalid command: unused arguments: ['mirror/image_32'] 2023-08-26T13:56:42.325 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:rbd mirror status : get status for rbd mirror 2023-08-26T13:56:42.325 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:admin_socket: invalid command 2023-08-26T13:56:42.326 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-08-26T13:56:42.326 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ sleep 8 2023-08-26T13:56:50.327 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ '[' cluster1 '!=' cluster1 ']' 2023-08-26T13:56:50.327 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ seq 0 3 2023-08-26T13:56:50.330 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-08-26T13:56:50.330 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_32 2023-08-26T13:56:50.331 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local cluster=cluster1:0 2023-08-26T13:56:50.331 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ shift 2023-08-26T13:56:50.331 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local instance 2023-08-26T13:56:50.331 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ set_cluster_instance cluster1:0 cluster instance 2023-08-26T13:56:50.331 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local val=cluster1:0 2023-08-26T13:56:50.332 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local cluster_var_name=cluster 2023-08-26T13:56:50.332 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local instance_var_name=instance 2023-08-26T13:56:50.332 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ cluster=cluster1 2023-08-26T13:56:50.332 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ instance=0 2023-08-26T13:56:50.332 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ '[' 0 = cluster1:0 ']' 2023-08-26T13:56:50.333 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ eval cluster=cluster1 2023-08-26T13:56:50.333 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ cluster=cluster1 2023-08-26T13:56:50.333 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ eval instance=0 2023-08-26T13:56:50.333 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ instance=0 2023-08-26T13:56:50.333 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ daemon_asok_file cluster1:0 cluster1 2023-08-26T13:56:50.333 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ local local_cluster=cluster1:0 2023-08-26T13:56:50.334 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ local cluster=cluster1 2023-08-26T13:56:50.334 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ local instance 2023-08-26T13:56:50.334 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2023-08-26T13:56:50.334 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ local val=cluster1:0 2023-08-26T13:56:50.334 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ local cluster_var_name=local_cluster 2023-08-26T13:56:50.335 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ local instance_var_name=instance 2023-08-26T13:56:50.335 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ cluster=cluster1 2023-08-26T13:56:50.335 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ instance=0 2023-08-26T13:56:50.335 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ '[' 0 = cluster1:0 ']' 2023-08-26T13:56:50.335 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ eval local_cluster=cluster1 2023-08-26T13:56:50.336 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++++ local_cluster=cluster1 2023-08-26T13:56:50.336 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ eval instance=0 2023-08-26T13:56:50.336 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++++ instance=0 2023-08-26T13:56:50.336 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2023-08-26T13:56:50.351 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-08-26T13:56:50.352 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-08-26T13:56:50.352 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-08-26T13:56:50.352 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_32 2023-08-26T13:56:50.453 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:Invalid command: unused arguments: ['mirror/image_32'] 2023-08-26T13:56:50.453 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:rbd mirror status : get status for rbd mirror 2023-08-26T13:56:50.453 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:admin_socket: invalid command 2023-08-26T13:56:50.456 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-08-26T13:56:50.456 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_32 2023-08-26T13:56:50.456 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local cluster=cluster1:1 2023-08-26T13:56:50.456 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ shift 2023-08-26T13:56:50.457 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local instance 2023-08-26T13:56:50.457 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ set_cluster_instance cluster1:1 cluster instance 2023-08-26T13:56:50.457 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local val=cluster1:1 2023-08-26T13:56:50.457 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local cluster_var_name=cluster 2023-08-26T13:56:50.457 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local instance_var_name=instance 2023-08-26T13:56:50.457 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ cluster=cluster1 2023-08-26T13:56:50.458 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ instance=1 2023-08-26T13:56:50.458 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ '[' 1 = cluster1:1 ']' 2023-08-26T13:56:50.458 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ eval cluster=cluster1 2023-08-26T13:56:50.458 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ cluster=cluster1 2023-08-26T13:56:50.458 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ eval instance=1 2023-08-26T13:56:50.458 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ instance=1 2023-08-26T13:56:50.459 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ daemon_asok_file cluster1:1 cluster1 2023-08-26T13:56:50.459 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ local local_cluster=cluster1:1 2023-08-26T13:56:50.459 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ local cluster=cluster1 2023-08-26T13:56:50.459 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ local instance 2023-08-26T13:56:50.459 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2023-08-26T13:56:50.460 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ local val=cluster1:1 2023-08-26T13:56:50.460 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ local cluster_var_name=local_cluster 2023-08-26T13:56:50.460 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ local instance_var_name=instance 2023-08-26T13:56:50.460 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ cluster=cluster1 2023-08-26T13:56:50.460 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ instance=1 2023-08-26T13:56:50.461 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ '[' 1 = cluster1:1 ']' 2023-08-26T13:56:50.461 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ eval local_cluster=cluster1 2023-08-26T13:56:50.461 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++++ local_cluster=cluster1 2023-08-26T13:56:50.461 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ eval instance=1 2023-08-26T13:56:50.461 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++++ instance=1 2023-08-26T13:56:50.461 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2023-08-26T13:56:50.475 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-08-26T13:56:50.476 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-08-26T13:56:50.476 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-08-26T13:56:50.476 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_32 2023-08-26T13:56:50.580 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:Invalid command: unused arguments: ['mirror/image_32'] 2023-08-26T13:56:50.580 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:rbd mirror status : get status for rbd mirror 2023-08-26T13:56:50.580 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:admin_socket: invalid command 2023-08-26T13:56:50.583 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-08-26T13:56:50.583 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/image_32 2023-08-26T13:56:50.583 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local cluster=cluster1:2 2023-08-26T13:56:50.584 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ shift 2023-08-26T13:56:50.584 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local instance 2023-08-26T13:56:50.584 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ set_cluster_instance cluster1:2 cluster instance 2023-08-26T13:56:50.584 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local val=cluster1:2 2023-08-26T13:56:50.584 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local cluster_var_name=cluster 2023-08-26T13:56:50.585 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local instance_var_name=instance 2023-08-26T13:56:50.585 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ cluster=cluster1 2023-08-26T13:56:50.585 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ instance=2 2023-08-26T13:56:50.585 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ '[' 2 = cluster1:2 ']' 2023-08-26T13:56:50.585 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ eval cluster=cluster1 2023-08-26T13:56:50.585 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ cluster=cluster1 2023-08-26T13:56:50.586 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ eval instance=2 2023-08-26T13:56:50.586 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ instance=2 2023-08-26T13:56:50.586 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ daemon_asok_file cluster1:2 cluster1 2023-08-26T13:56:50.586 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ local local_cluster=cluster1:2 2023-08-26T13:56:50.586 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ local cluster=cluster1 2023-08-26T13:56:50.586 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ local instance 2023-08-26T13:56:50.587 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2023-08-26T13:56:50.587 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ local val=cluster1:2 2023-08-26T13:56:50.587 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ local cluster_var_name=local_cluster 2023-08-26T13:56:50.587 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ local instance_var_name=instance 2023-08-26T13:56:50.587 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ cluster=cluster1 2023-08-26T13:56:50.588 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ instance=2 2023-08-26T13:56:50.588 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ '[' 2 = cluster1:2 ']' 2023-08-26T13:56:50.588 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ eval local_cluster=cluster1 2023-08-26T13:56:50.588 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++++ local_cluster=cluster1 2023-08-26T13:56:50.588 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ eval instance=2 2023-08-26T13:56:50.589 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++++ instance=2 2023-08-26T13:56:50.589 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2023-08-26T13:56:50.601 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-08-26T13:56:50.601 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-08-26T13:56:50.601 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-08-26T13:56:50.601 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/image_32 2023-08-26T13:56:50.709 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:Invalid command: unused arguments: ['mirror/image_32'] 2023-08-26T13:56:50.710 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:rbd mirror status : get status for rbd mirror 2023-08-26T13:56:50.710 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:admin_socket: invalid command 2023-08-26T13:56:50.713 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-08-26T13:56:50.713 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ admin_daemon cluster1:3 rbd mirror status mirror/image_32 2023-08-26T13:56:50.713 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local cluster=cluster1:3 2023-08-26T13:56:50.713 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ shift 2023-08-26T13:56:50.713 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local instance 2023-08-26T13:56:50.714 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ set_cluster_instance cluster1:3 cluster instance 2023-08-26T13:56:50.714 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local val=cluster1:3 2023-08-26T13:56:50.714 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local cluster_var_name=cluster 2023-08-26T13:56:50.714 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local instance_var_name=instance 2023-08-26T13:56:50.714 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ cluster=cluster1 2023-08-26T13:56:50.714 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ instance=3 2023-08-26T13:56:50.715 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ '[' 3 = cluster1:3 ']' 2023-08-26T13:56:50.715 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ eval cluster=cluster1 2023-08-26T13:56:50.715 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ cluster=cluster1 2023-08-26T13:56:50.715 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ eval instance=3 2023-08-26T13:56:50.715 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ instance=3 2023-08-26T13:56:50.716 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ daemon_asok_file cluster1:3 cluster1 2023-08-26T13:56:50.716 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ local local_cluster=cluster1:3 2023-08-26T13:56:50.716 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ local cluster=cluster1 2023-08-26T13:56:50.716 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ local instance 2023-08-26T13:56:50.716 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ set_cluster_instance cluster1:3 local_cluster instance 2023-08-26T13:56:50.717 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ local val=cluster1:3 2023-08-26T13:56:50.717 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ local cluster_var_name=local_cluster 2023-08-26T13:56:50.717 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ local instance_var_name=instance 2023-08-26T13:56:50.717 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ cluster=cluster1 2023-08-26T13:56:50.717 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ instance=3 2023-08-26T13:56:50.717 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ '[' 3 = cluster1:3 ']' 2023-08-26T13:56:50.718 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ eval local_cluster=cluster1 2023-08-26T13:56:50.718 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++++ local_cluster=cluster1 2023-08-26T13:56:50.718 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ eval instance=3 2023-08-26T13:56:50.718 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++++ instance=3 2023-08-26T13:56:50.718 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2023-08-26T13:56:50.732 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2023-08-26T13:56:50.732 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2023-08-26T13:56:50.732 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2023-08-26T13:56:50.732 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror status mirror/image_32 2023-08-26T13:56:50.836 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:Invalid command: unused arguments: ['mirror/image_32'] 2023-08-26T13:56:50.837 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:rbd mirror status : get status for rbd mirror 2023-08-26T13:56:50.837 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:admin_socket: invalid command 2023-08-26T13:56:50.837 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-08-26T13:56:50.838 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ sleep 8 2023-08-26T13:56:58.839 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ '[' cluster1 '!=' cluster1 ']' 2023-08-26T13:56:58.840 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ seq 0 3 2023-08-26T13:56:58.845 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-08-26T13:56:58.845 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_32 2023-08-26T13:56:58.846 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local cluster=cluster1:0 2023-08-26T13:56:58.846 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ shift 2023-08-26T13:56:58.846 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local instance 2023-08-26T13:56:58.847 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ set_cluster_instance cluster1:0 cluster instance 2023-08-26T13:56:58.847 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local val=cluster1:0 2023-08-26T13:56:58.847 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local cluster_var_name=cluster 2023-08-26T13:56:58.848 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local instance_var_name=instance 2023-08-26T13:56:58.848 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ cluster=cluster1 2023-08-26T13:56:58.848 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ instance=0 2023-08-26T13:56:58.849 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ '[' 0 = cluster1:0 ']' 2023-08-26T13:56:58.849 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ eval cluster=cluster1 2023-08-26T13:56:58.849 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ cluster=cluster1 2023-08-26T13:56:58.850 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ eval instance=0 2023-08-26T13:56:58.850 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ instance=0 2023-08-26T13:56:58.850 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ daemon_asok_file cluster1:0 cluster1 2023-08-26T13:56:58.851 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ local local_cluster=cluster1:0 2023-08-26T13:56:58.851 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ local cluster=cluster1 2023-08-26T13:56:58.851 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ local instance 2023-08-26T13:56:58.852 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2023-08-26T13:56:58.852 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ local val=cluster1:0 2023-08-26T13:56:58.852 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ local cluster_var_name=local_cluster 2023-08-26T13:56:58.853 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ local instance_var_name=instance 2023-08-26T13:56:58.853 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ cluster=cluster1 2023-08-26T13:56:58.853 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ instance=0 2023-08-26T13:56:58.854 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ '[' 0 = cluster1:0 ']' 2023-08-26T13:56:58.854 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ eval local_cluster=cluster1 2023-08-26T13:56:58.854 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++++ local_cluster=cluster1 2023-08-26T13:56:58.855 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ eval instance=0 2023-08-26T13:56:58.855 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++++ instance=0 2023-08-26T13:56:58.855 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2023-08-26T13:56:58.864 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-08-26T13:56:58.864 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-08-26T13:56:58.865 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-08-26T13:56:58.865 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_32 2023-08-26T13:56:58.988 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ return 0 2023-08-26T13:56:58.989 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ status_result=' "state": "Replaying"' 2023-08-26T13:56:58.989 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ echo ' "state": "Replaying"' 2023-08-26T13:56:58.990 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ grep -i Replaying 2023-08-26T13:56:58.990 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: "state": "Replaying" 2023-08-26T13:56:58.992 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ current_state=started 2023-08-26T13:56:58.992 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ test started = started 2023-08-26T13:56:58.992 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ return 0 2023-08-26T13:56:58.993 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_32 2023-08-26T13:56:58.993 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local local_cluster=cluster1 2023-08-26T13:56:58.993 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster2 2023-08-26T13:56:58.993 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T13:56:58.993 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=image_32 2023-08-26T13:56:58.994 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ '[' snapshot = journal ']' 2023-08-26T13:56:58.994 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ '[' snapshot = snapshot ']' 2023-08-26T13:56:58.994 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_32 2023-08-26T13:56:58.994 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local local_cluster=cluster1 2023-08-26T13:56:58.994 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster2 2023-08-26T13:56:58.995 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T13:56:58.995 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=image_32 2023-08-26T13:56:58.995 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ mkfname cluster2-mirror-image_32.status 2023-08-26T13:56:58.995 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ echo cluster2-mirror-image_32.status 2023-08-26T13:56:58.995 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ sed -e 's|[/ ]|_|g' 2023-08-26T13:56:58.996 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local status_log=/tmp/tmp.TfDn6BcQ3N/cluster2-mirror-image_32.status 2023-08-26T13:56:58.996 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ mkfname cluster1-mirror-image_32.status 2023-08-26T13:56:58.997 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ echo cluster1-mirror-image_32.status 2023-08-26T13:56:58.997 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ sed -e 's|[/ ]|_|g' 2023-08-26T13:56:58.997 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local local_status_log=/tmp/tmp.TfDn6BcQ3N/cluster1-mirror-image_32.status 2023-08-26T13:56:58.998 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ mirror_image_snapshot cluster2 mirror image_32 2023-08-26T13:56:58.998 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster2 2023-08-26T13:56:58.998 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T13:56:58.998 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=image_32 2023-08-26T13:56:58.998 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_32 2023-08-26T13:56:59.267 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:Snapshot ID: 181 2023-08-26T13:56:59.275 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_32 /tmp/tmp.TfDn6BcQ3N/cluster2-mirror-image_32.status 2023-08-26T13:56:59.276 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster2 2023-08-26T13:56:59.276 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T13:56:59.276 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=image_32 2023-08-26T13:56:59.276 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local log=/tmp/tmp.TfDn6BcQ3N/cluster2-mirror-image_32.status 2023-08-26T13:56:59.276 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_32 --format xml 2023-08-26T13:56:59.277 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-08-26T13:56:59.318 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ xmlstarlet sel -t -v //snapshot/id 2023-08-26T13:56:59.320 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local snapshot_id=181 2023-08-26T13:56:59.320 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ true 2023-08-26T13:56:59.320 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-08-26T13:56:59.320 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ sleep 0.2 2023-08-26T13:56:59.521 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_32 /tmp/tmp.TfDn6BcQ3N/cluster1-mirror-image_32.status 2023-08-26T13:56:59.521 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster1 2023-08-26T13:56:59.521 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T13:56:59.521 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=image_32 2023-08-26T13:56:59.522 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local log=/tmp/tmp.TfDn6BcQ3N/cluster1-mirror-image_32.status 2023-08-26T13:56:59.522 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_32 --format xml 2023-08-26T13:56:59.522 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-08-26T13:56:59.567 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ true 2023-08-26T13:56:59.567 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-08-26T13:56:59.568 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:-:1.1: Document is empty 2023-08-26T13:56:59.568 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr: 2023-08-26T13:56:59.569 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:^ 2023-08-26T13:56:59.569 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local primary_snapshot_id= 2023-08-26T13:56:59.569 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ test 181 = '' 2023-08-26T13:56:59.570 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-08-26T13:56:59.570 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ sleep 0.4 2023-08-26T13:56:59.970 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_32 /tmp/tmp.TfDn6BcQ3N/cluster1-mirror-image_32.status 2023-08-26T13:56:59.971 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster1 2023-08-26T13:56:59.971 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T13:56:59.971 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=image_32 2023-08-26T13:56:59.971 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local log=/tmp/tmp.TfDn6BcQ3N/cluster1-mirror-image_32.status 2023-08-26T13:56:59.971 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_32 --format xml 2023-08-26T13:56:59.972 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-08-26T13:57:00.016 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ true 2023-08-26T13:57:00.017 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-08-26T13:57:00.018 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:-:1.1: Document is empty 2023-08-26T13:57:00.018 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr: 2023-08-26T13:57:00.018 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:^ 2023-08-26T13:57:00.019 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local primary_snapshot_id= 2023-08-26T13:57:00.019 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ test 181 = '' 2023-08-26T13:57:00.019 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-08-26T13:57:00.019 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ sleep 0.8 2023-08-26T13:57:00.820 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_32 /tmp/tmp.TfDn6BcQ3N/cluster1-mirror-image_32.status 2023-08-26T13:57:00.820 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster1 2023-08-26T13:57:00.820 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T13:57:00.820 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=image_32 2023-08-26T13:57:00.820 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local log=/tmp/tmp.TfDn6BcQ3N/cluster1-mirror-image_32.status 2023-08-26T13:57:00.821 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_32 --format xml 2023-08-26T13:57:00.821 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-08-26T13:57:00.864 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-08-26T13:57:00.866 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local primary_snapshot_id=181 2023-08-26T13:57:00.867 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ test 181 = 181 2023-08-26T13:57:00.867 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ return 0 2023-08-26T13:57:00.867 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ wait_for_snap_present cluster1 mirror image_32 snap 2023-08-26T13:57:00.867 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster1 2023-08-26T13:57:00.867 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T13:57:00.868 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=image_32 2023-08-26T13:57:00.868 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local snap_name=snap 2023-08-26T13:57:00.868 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local s 2023-08-26T13:57:00.868 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2023-08-26T13:57:00.868 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ sleep 1 2023-08-26T13:57:01.867 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rbd --cluster cluster1 info mirror/image_32@snap 2023-08-26T13:57:01.918 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:rbd image 'image_32': 2023-08-26T13:57:01.918 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: size 128 MiB in 32 objects 2023-08-26T13:57:01.918 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: order 22 (4 MiB objects) 2023-08-26T13:57:01.919 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: snapshot_count: 2 2023-08-26T13:57:01.919 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: id: 12cf8cc132b 2023-08-26T13:57:01.919 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: block_name_prefix: rbd_data.12cf8cc132b 2023-08-26T13:57:01.919 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: format: 2 2023-08-26T13:57:01.919 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: features: layering, exclusive-lock, non-primary 2023-08-26T13:57:01.920 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: op_features: 2023-08-26T13:57:01.920 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: flags: 2023-08-26T13:57:01.920 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: create_timestamp: Sat Aug 26 13:48:00 2023 2023-08-26T13:57:01.920 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: access_timestamp: Sat Aug 26 13:48:00 2023 2023-08-26T13:57:01.920 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: modify_timestamp: Sat Aug 26 13:48:00 2023 2023-08-26T13:57:01.921 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: protected: False 2023-08-26T13:57:01.921 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: mirroring state: enabled 2023-08-26T13:57:01.921 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: mirroring mode: snapshot 2023-08-26T13:57:01.921 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: mirroring global id: 9ec01316-ea12-44fa-bbc4-3dc34cb1f619 2023-08-26T13:57:01.921 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: mirroring primary: false 2023-08-26T13:57:01.923 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ return 0 2023-08-26T13:57:01.923 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ compare_image_snaps mirror image_32 snap 2023-08-26T13:57:01.923 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T13:57:01.923 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=image_32 2023-08-26T13:57:01.923 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local snap_name=snap 2023-08-26T13:57:01.924 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local ret=0 2023-08-26T13:57:01.924 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local rmt_export=/tmp/tmp.TfDn6BcQ3N/cluster2-mirror-image_32.export 2023-08-26T13:57:01.924 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local loc_export=/tmp/tmp.TfDn6BcQ3N/cluster1-mirror-image_32.export 2023-08-26T13:57:01.924 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rm -f /tmp/tmp.TfDn6BcQ3N/cluster2-mirror-image_32.export /tmp/tmp.TfDn6BcQ3N/cluster1-mirror-image_32.export 2023-08-26T13:57:01.924 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rbd --cluster cluster2 -p mirror export image_32@snap /tmp/tmp.TfDn6BcQ3N/cluster2-mirror-image_32.export 2023-08-26T13:57:02.110 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-08-26T13:57:02.116 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rbd --cluster cluster1 -p mirror export image_32@snap /tmp/tmp.TfDn6BcQ3N/cluster1-mirror-image_32.export 2023-08-26T13:57:02.358 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-08-26T13:57:02.364 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ cmp /tmp/tmp.TfDn6BcQ3N/cluster2-mirror-image_32.export /tmp/tmp.TfDn6BcQ3N/cluster1-mirror-image_32.export 2023-08-26T13:57:02.419 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rm -f /tmp/tmp.TfDn6BcQ3N/cluster2-mirror-image_32.export /tmp/tmp.TfDn6BcQ3N/cluster1-mirror-image_32.export 2023-08-26T13:57:02.456 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ return 0 2023-08-26T13:57:02.457 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-08-26T13:57:02.457 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ image=image_33 2023-08-26T13:57:02.457 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ create_snap cluster2 mirror image_33 snap 2023-08-26T13:57:02.457 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster2 2023-08-26T13:57:02.457 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T13:57:02.458 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=image_33 2023-08-26T13:57:02.458 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local snap_name=snap 2023-08-26T13:57:02.458 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rbd --cluster cluster2 -p mirror snap create image_33@snap --debug-rbd=20 --debug-journaler=20 2023-08-26T13:57:04.292 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ wait_for_image_replay_started cluster1 mirror image_33 2023-08-26T13:57:04.292 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster1 2023-08-26T13:57:04.292 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T13:57:04.293 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=image_33 2023-08-26T13:57:04.293 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ wait_for_image_replay_state cluster1 mirror image_33 started 2023-08-26T13:57:04.293 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster1 2023-08-26T13:57:04.293 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T13:57:04.293 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=image_33 2023-08-26T13:57:04.294 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local state=started 2023-08-26T13:57:04.294 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local s 2023-08-26T13:57:04.294 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-08-26T13:57:04.294 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ sleep 1 2023-08-26T13:57:05.293 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ test_image_replay_state cluster1 mirror image_33 started 2023-08-26T13:57:05.294 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster1 2023-08-26T13:57:05.294 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T13:57:05.294 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=image_33 2023-08-26T13:57:05.294 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local test_state=started 2023-08-26T13:57:05.295 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local status_result 2023-08-26T13:57:05.295 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local current_state=stopped 2023-08-26T13:57:05.298 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_33 2023-08-26T13:57:05.299 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ grep -i state 2023-08-26T13:57:05.299 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local cluster_instance=cluster1 2023-08-26T13:57:05.299 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ shift 2023-08-26T13:57:05.299 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local cluster=cluster1 2023-08-26T13:57:05.300 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local instance=cluster1 2023-08-26T13:57:05.300 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local loop_instance 2023-08-26T13:57:05.300 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-08-26T13:57:05.300 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ sleep 0 2023-08-26T13:57:05.300 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ '[' cluster1 '!=' cluster1 ']' 2023-08-26T13:57:05.301 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ seq 0 3 2023-08-26T13:57:05.301 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-08-26T13:57:05.301 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_33 2023-08-26T13:57:05.301 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local cluster=cluster1:0 2023-08-26T13:57:05.301 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ shift 2023-08-26T13:57:05.301 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local instance 2023-08-26T13:57:05.302 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ set_cluster_instance cluster1:0 cluster instance 2023-08-26T13:57:05.302 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local val=cluster1:0 2023-08-26T13:57:05.302 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local cluster_var_name=cluster 2023-08-26T13:57:05.302 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local instance_var_name=instance 2023-08-26T13:57:05.302 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ cluster=cluster1 2023-08-26T13:57:05.303 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ instance=0 2023-08-26T13:57:05.303 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ '[' 0 = cluster1:0 ']' 2023-08-26T13:57:05.303 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ eval cluster=cluster1 2023-08-26T13:57:05.303 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ cluster=cluster1 2023-08-26T13:57:05.303 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ eval instance=0 2023-08-26T13:57:05.304 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ instance=0 2023-08-26T13:57:05.304 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ daemon_asok_file cluster1:0 cluster1 2023-08-26T13:57:05.304 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ local local_cluster=cluster1:0 2023-08-26T13:57:05.304 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ local cluster=cluster1 2023-08-26T13:57:05.304 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ local instance 2023-08-26T13:57:05.304 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2023-08-26T13:57:05.305 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ local val=cluster1:0 2023-08-26T13:57:05.305 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ local cluster_var_name=local_cluster 2023-08-26T13:57:05.305 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ local instance_var_name=instance 2023-08-26T13:57:05.305 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ cluster=cluster1 2023-08-26T13:57:05.305 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ instance=0 2023-08-26T13:57:05.306 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ '[' 0 = cluster1:0 ']' 2023-08-26T13:57:05.306 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ eval local_cluster=cluster1 2023-08-26T13:57:05.306 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++++ local_cluster=cluster1 2023-08-26T13:57:05.306 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ eval instance=0 2023-08-26T13:57:05.306 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++++ instance=0 2023-08-26T13:57:05.307 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2023-08-26T13:57:05.320 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-08-26T13:57:05.320 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-08-26T13:57:05.321 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-08-26T13:57:05.321 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_33 2023-08-26T13:57:05.432 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:Invalid command: unused arguments: ['mirror/image_33'] 2023-08-26T13:57:05.433 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:rbd mirror status : get status for rbd mirror 2023-08-26T13:57:05.433 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:admin_socket: invalid command 2023-08-26T13:57:05.435 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-08-26T13:57:05.436 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_33 2023-08-26T13:57:05.436 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local cluster=cluster1:1 2023-08-26T13:57:05.436 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ shift 2023-08-26T13:57:05.436 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local instance 2023-08-26T13:57:05.436 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ set_cluster_instance cluster1:1 cluster instance 2023-08-26T13:57:05.437 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local val=cluster1:1 2023-08-26T13:57:05.437 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local cluster_var_name=cluster 2023-08-26T13:57:05.437 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local instance_var_name=instance 2023-08-26T13:57:05.439 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ cluster=cluster1 2023-08-26T13:57:05.439 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ instance=1 2023-08-26T13:57:05.439 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ '[' 1 = cluster1:1 ']' 2023-08-26T13:57:05.440 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ eval cluster=cluster1 2023-08-26T13:57:05.440 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ cluster=cluster1 2023-08-26T13:57:05.441 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ eval instance=1 2023-08-26T13:57:05.441 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ instance=1 2023-08-26T13:57:05.442 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ daemon_asok_file cluster1:1 cluster1 2023-08-26T13:57:05.442 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ local local_cluster=cluster1:1 2023-08-26T13:57:05.442 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ local cluster=cluster1 2023-08-26T13:57:05.443 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ local instance 2023-08-26T13:57:05.443 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2023-08-26T13:57:05.444 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ local val=cluster1:1 2023-08-26T13:57:05.444 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ local cluster_var_name=local_cluster 2023-08-26T13:57:05.444 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ local instance_var_name=instance 2023-08-26T13:57:05.444 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ cluster=cluster1 2023-08-26T13:57:05.445 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ instance=1 2023-08-26T13:57:05.445 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ '[' 1 = cluster1:1 ']' 2023-08-26T13:57:05.445 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ eval local_cluster=cluster1 2023-08-26T13:57:05.445 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++++ local_cluster=cluster1 2023-08-26T13:57:05.445 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ eval instance=1 2023-08-26T13:57:05.446 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++++ instance=1 2023-08-26T13:57:05.446 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2023-08-26T13:57:05.454 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-08-26T13:57:05.454 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-08-26T13:57:05.455 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-08-26T13:57:05.455 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_33 2023-08-26T13:57:05.561 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:Invalid command: unused arguments: ['mirror/image_33'] 2023-08-26T13:57:05.561 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:rbd mirror status : get status for rbd mirror 2023-08-26T13:57:05.561 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:admin_socket: invalid command 2023-08-26T13:57:05.564 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-08-26T13:57:05.565 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/image_33 2023-08-26T13:57:05.565 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local cluster=cluster1:2 2023-08-26T13:57:05.565 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ shift 2023-08-26T13:57:05.565 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local instance 2023-08-26T13:57:05.565 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ set_cluster_instance cluster1:2 cluster instance 2023-08-26T13:57:05.566 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local val=cluster1:2 2023-08-26T13:57:05.566 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local cluster_var_name=cluster 2023-08-26T13:57:05.566 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local instance_var_name=instance 2023-08-26T13:57:05.566 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ cluster=cluster1 2023-08-26T13:57:05.566 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ instance=2 2023-08-26T13:57:05.567 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ '[' 2 = cluster1:2 ']' 2023-08-26T13:57:05.567 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ eval cluster=cluster1 2023-08-26T13:57:05.567 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ cluster=cluster1 2023-08-26T13:57:05.567 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ eval instance=2 2023-08-26T13:57:05.567 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ instance=2 2023-08-26T13:57:05.568 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ daemon_asok_file cluster1:2 cluster1 2023-08-26T13:57:05.568 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ local local_cluster=cluster1:2 2023-08-26T13:57:05.568 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ local cluster=cluster1 2023-08-26T13:57:05.568 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ local instance 2023-08-26T13:57:05.568 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2023-08-26T13:57:05.569 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ local val=cluster1:2 2023-08-26T13:57:05.569 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ local cluster_var_name=local_cluster 2023-08-26T13:57:05.569 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ local instance_var_name=instance 2023-08-26T13:57:05.569 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ cluster=cluster1 2023-08-26T13:57:05.569 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ instance=2 2023-08-26T13:57:05.570 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ '[' 2 = cluster1:2 ']' 2023-08-26T13:57:05.570 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ eval local_cluster=cluster1 2023-08-26T13:57:05.570 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++++ local_cluster=cluster1 2023-08-26T13:57:05.570 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ eval instance=2 2023-08-26T13:57:05.570 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++++ instance=2 2023-08-26T13:57:05.570 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2023-08-26T13:57:05.584 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-08-26T13:57:05.584 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-08-26T13:57:05.584 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-08-26T13:57:05.584 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/image_33 2023-08-26T13:57:05.689 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:Invalid command: unused arguments: ['mirror/image_33'] 2023-08-26T13:57:05.689 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:rbd mirror status : get status for rbd mirror 2023-08-26T13:57:05.689 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:admin_socket: invalid command 2023-08-26T13:57:05.692 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-08-26T13:57:05.692 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ admin_daemon cluster1:3 rbd mirror status mirror/image_33 2023-08-26T13:57:05.693 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local cluster=cluster1:3 2023-08-26T13:57:05.693 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ shift 2023-08-26T13:57:05.693 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local instance 2023-08-26T13:57:05.693 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ set_cluster_instance cluster1:3 cluster instance 2023-08-26T13:57:05.694 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local val=cluster1:3 2023-08-26T13:57:05.694 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local cluster_var_name=cluster 2023-08-26T13:57:05.694 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local instance_var_name=instance 2023-08-26T13:57:05.694 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ cluster=cluster1 2023-08-26T13:57:05.694 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ instance=3 2023-08-26T13:57:05.695 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ '[' 3 = cluster1:3 ']' 2023-08-26T13:57:05.695 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ eval cluster=cluster1 2023-08-26T13:57:05.695 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ cluster=cluster1 2023-08-26T13:57:05.695 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ eval instance=3 2023-08-26T13:57:05.695 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ instance=3 2023-08-26T13:57:05.696 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ daemon_asok_file cluster1:3 cluster1 2023-08-26T13:57:05.696 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ local local_cluster=cluster1:3 2023-08-26T13:57:05.696 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ local cluster=cluster1 2023-08-26T13:57:05.696 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ local instance 2023-08-26T13:57:05.696 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ set_cluster_instance cluster1:3 local_cluster instance 2023-08-26T13:57:05.697 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ local val=cluster1:3 2023-08-26T13:57:05.697 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ local cluster_var_name=local_cluster 2023-08-26T13:57:05.697 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ local instance_var_name=instance 2023-08-26T13:57:05.697 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ cluster=cluster1 2023-08-26T13:57:05.697 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ instance=3 2023-08-26T13:57:05.698 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ '[' 3 = cluster1:3 ']' 2023-08-26T13:57:05.698 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ eval local_cluster=cluster1 2023-08-26T13:57:05.698 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++++ local_cluster=cluster1 2023-08-26T13:57:05.698 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ eval instance=3 2023-08-26T13:57:05.698 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++++ instance=3 2023-08-26T13:57:05.698 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2023-08-26T13:57:05.711 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2023-08-26T13:57:05.711 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2023-08-26T13:57:05.711 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2023-08-26T13:57:05.712 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror status mirror/image_33 2023-08-26T13:57:05.826 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ return 0 2023-08-26T13:57:05.826 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ status_result=' "state": "Replaying"' 2023-08-26T13:57:05.827 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ echo ' "state": "Replaying"' 2023-08-26T13:57:05.827 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ grep -i Replaying 2023-08-26T13:57:05.827 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: "state": "Replaying" 2023-08-26T13:57:05.829 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ current_state=started 2023-08-26T13:57:05.830 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ test started = started 2023-08-26T13:57:05.830 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ return 0 2023-08-26T13:57:05.830 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_33 2023-08-26T13:57:05.830 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local local_cluster=cluster1 2023-08-26T13:57:05.830 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster2 2023-08-26T13:57:05.831 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T13:57:05.831 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=image_33 2023-08-26T13:57:05.831 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ '[' snapshot = journal ']' 2023-08-26T13:57:05.831 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ '[' snapshot = snapshot ']' 2023-08-26T13:57:05.831 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_33 2023-08-26T13:57:05.832 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local local_cluster=cluster1 2023-08-26T13:57:05.832 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster2 2023-08-26T13:57:05.832 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T13:57:05.833 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=image_33 2023-08-26T13:57:05.833 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ mkfname cluster2-mirror-image_33.status 2023-08-26T13:57:05.833 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ echo cluster2-mirror-image_33.status 2023-08-26T13:57:05.833 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ sed -e 's|[/ ]|_|g' 2023-08-26T13:57:05.833 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local status_log=/tmp/tmp.TfDn6BcQ3N/cluster2-mirror-image_33.status 2023-08-26T13:57:05.834 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ mkfname cluster1-mirror-image_33.status 2023-08-26T13:57:05.834 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ echo cluster1-mirror-image_33.status 2023-08-26T13:57:05.834 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ sed -e 's|[/ ]|_|g' 2023-08-26T13:57:05.835 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local local_status_log=/tmp/tmp.TfDn6BcQ3N/cluster1-mirror-image_33.status 2023-08-26T13:57:05.835 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ mirror_image_snapshot cluster2 mirror image_33 2023-08-26T13:57:05.835 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster2 2023-08-26T13:57:05.835 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T13:57:05.835 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=image_33 2023-08-26T13:57:05.836 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_33 2023-08-26T13:57:06.304 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:Snapshot ID: 183 2023-08-26T13:57:06.313 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_33 /tmp/tmp.TfDn6BcQ3N/cluster2-mirror-image_33.status 2023-08-26T13:57:06.314 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster2 2023-08-26T13:57:06.314 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T13:57:06.314 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=image_33 2023-08-26T13:57:06.314 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local log=/tmp/tmp.TfDn6BcQ3N/cluster2-mirror-image_33.status 2023-08-26T13:57:06.314 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_33 --format xml 2023-08-26T13:57:06.315 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-08-26T13:57:06.356 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ xmlstarlet sel -t -v //snapshot/id 2023-08-26T13:57:06.358 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local snapshot_id=183 2023-08-26T13:57:06.358 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ true 2023-08-26T13:57:06.359 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-08-26T13:57:06.359 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ sleep 0.2 2023-08-26T13:57:06.560 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_33 /tmp/tmp.TfDn6BcQ3N/cluster1-mirror-image_33.status 2023-08-26T13:57:06.561 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster1 2023-08-26T13:57:06.561 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T13:57:06.561 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=image_33 2023-08-26T13:57:06.562 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local log=/tmp/tmp.TfDn6BcQ3N/cluster1-mirror-image_33.status 2023-08-26T13:57:06.562 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_33 --format xml 2023-08-26T13:57:06.562 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-08-26T13:57:06.607 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ true 2023-08-26T13:57:06.608 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-08-26T13:57:06.609 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:-:1.1: Document is empty 2023-08-26T13:57:06.609 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr: 2023-08-26T13:57:06.609 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:^ 2023-08-26T13:57:06.610 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local primary_snapshot_id= 2023-08-26T13:57:06.610 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ test 183 = '' 2023-08-26T13:57:06.610 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-08-26T13:57:06.611 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ sleep 0.4 2023-08-26T13:57:07.011 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_33 /tmp/tmp.TfDn6BcQ3N/cluster1-mirror-image_33.status 2023-08-26T13:57:07.011 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster1 2023-08-26T13:57:07.011 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T13:57:07.012 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=image_33 2023-08-26T13:57:07.012 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local log=/tmp/tmp.TfDn6BcQ3N/cluster1-mirror-image_33.status 2023-08-26T13:57:07.012 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_33 --format xml 2023-08-26T13:57:07.012 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-08-26T13:57:07.059 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ true 2023-08-26T13:57:07.059 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-08-26T13:57:07.060 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:-:1.1: Document is empty 2023-08-26T13:57:07.060 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr: 2023-08-26T13:57:07.060 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:^ 2023-08-26T13:57:07.061 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local primary_snapshot_id= 2023-08-26T13:57:07.061 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ test 183 = '' 2023-08-26T13:57:07.061 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-08-26T13:57:07.062 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ sleep 0.8 2023-08-26T13:57:07.862 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_33 /tmp/tmp.TfDn6BcQ3N/cluster1-mirror-image_33.status 2023-08-26T13:57:07.862 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster1 2023-08-26T13:57:07.862 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T13:57:07.863 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=image_33 2023-08-26T13:57:07.863 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local log=/tmp/tmp.TfDn6BcQ3N/cluster1-mirror-image_33.status 2023-08-26T13:57:07.863 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_33 --format xml 2023-08-26T13:57:07.863 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-08-26T13:57:07.909 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-08-26T13:57:07.911 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local primary_snapshot_id=183 2023-08-26T13:57:07.911 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ test 183 = 183 2023-08-26T13:57:07.911 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ return 0 2023-08-26T13:57:07.912 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ wait_for_snap_present cluster1 mirror image_33 snap 2023-08-26T13:57:07.912 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster1 2023-08-26T13:57:07.912 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T13:57:07.912 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=image_33 2023-08-26T13:57:07.912 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local snap_name=snap 2023-08-26T13:57:07.913 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local s 2023-08-26T13:57:07.913 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2023-08-26T13:57:07.913 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ sleep 1 2023-08-26T13:57:08.912 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rbd --cluster cluster1 info mirror/image_33@snap 2023-08-26T13:57:08.956 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:rbd image 'image_33': 2023-08-26T13:57:08.957 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: size 128 MiB in 32 objects 2023-08-26T13:57:08.957 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: order 22 (4 MiB objects) 2023-08-26T13:57:08.957 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: snapshot_count: 2 2023-08-26T13:57:08.957 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: id: 10b7695bbb68 2023-08-26T13:57:08.957 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: block_name_prefix: rbd_data.10b7695bbb68 2023-08-26T13:57:08.958 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: format: 2 2023-08-26T13:57:08.958 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: features: layering, exclusive-lock, non-primary 2023-08-26T13:57:08.958 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: op_features: 2023-08-26T13:57:08.958 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: flags: 2023-08-26T13:57:08.958 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: create_timestamp: Sat Aug 26 13:48:06 2023 2023-08-26T13:57:08.959 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: access_timestamp: Sat Aug 26 13:48:06 2023 2023-08-26T13:57:08.959 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: modify_timestamp: Sat Aug 26 13:48:06 2023 2023-08-26T13:57:08.959 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: protected: False 2023-08-26T13:57:08.959 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: mirroring state: enabled 2023-08-26T13:57:08.959 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: mirroring mode: snapshot 2023-08-26T13:57:08.960 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: mirroring global id: 3c30bb33-dd5f-4030-adb6-12b16331cd53 2023-08-26T13:57:08.960 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: mirroring primary: false 2023-08-26T13:57:08.961 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ return 0 2023-08-26T13:57:08.961 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ compare_image_snaps mirror image_33 snap 2023-08-26T13:57:08.961 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T13:57:08.962 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=image_33 2023-08-26T13:57:08.962 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local snap_name=snap 2023-08-26T13:57:08.962 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local ret=0 2023-08-26T13:57:08.963 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local rmt_export=/tmp/tmp.TfDn6BcQ3N/cluster2-mirror-image_33.export 2023-08-26T13:57:08.963 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local loc_export=/tmp/tmp.TfDn6BcQ3N/cluster1-mirror-image_33.export 2023-08-26T13:57:08.963 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rm -f /tmp/tmp.TfDn6BcQ3N/cluster2-mirror-image_33.export /tmp/tmp.TfDn6BcQ3N/cluster1-mirror-image_33.export 2023-08-26T13:57:08.963 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rbd --cluster cluster2 -p mirror export image_33@snap /tmp/tmp.TfDn6BcQ3N/cluster2-mirror-image_33.export 2023-08-26T13:57:09.146 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-08-26T13:57:09.152 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rbd --cluster cluster1 -p mirror export image_33@snap /tmp/tmp.TfDn6BcQ3N/cluster1-mirror-image_33.export 2023-08-26T13:57:09.388 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-08-26T13:57:09.395 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ cmp /tmp/tmp.TfDn6BcQ3N/cluster2-mirror-image_33.export /tmp/tmp.TfDn6BcQ3N/cluster1-mirror-image_33.export 2023-08-26T13:57:09.450 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rm -f /tmp/tmp.TfDn6BcQ3N/cluster2-mirror-image_33.export /tmp/tmp.TfDn6BcQ3N/cluster1-mirror-image_33.export 2023-08-26T13:57:09.487 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ return 0 2023-08-26T13:57:09.488 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-08-26T13:57:09.488 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ image=image_34 2023-08-26T13:57:09.488 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ create_snap cluster2 mirror image_34 snap 2023-08-26T13:57:09.488 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster2 2023-08-26T13:57:09.489 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T13:57:09.489 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=image_34 2023-08-26T13:57:09.489 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local snap_name=snap 2023-08-26T13:57:09.489 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rbd --cluster cluster2 -p mirror snap create image_34@snap --debug-rbd=20 --debug-journaler=20 2023-08-26T13:57:11.303 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ wait_for_image_replay_started cluster1 mirror image_34 2023-08-26T13:57:11.303 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster1 2023-08-26T13:57:11.303 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T13:57:11.304 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=image_34 2023-08-26T13:57:11.304 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ wait_for_image_replay_state cluster1 mirror image_34 started 2023-08-26T13:57:11.304 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster1 2023-08-26T13:57:11.304 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T13:57:11.304 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=image_34 2023-08-26T13:57:11.305 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local state=started 2023-08-26T13:57:11.305 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local s 2023-08-26T13:57:11.305 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-08-26T13:57:11.305 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ sleep 1 2023-08-26T13:57:12.304 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ test_image_replay_state cluster1 mirror image_34 started 2023-08-26T13:57:12.304 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster1 2023-08-26T13:57:12.305 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T13:57:12.305 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=image_34 2023-08-26T13:57:12.305 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local test_state=started 2023-08-26T13:57:12.305 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local status_result 2023-08-26T13:57:12.305 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local current_state=stopped 2023-08-26T13:57:12.306 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_34 2023-08-26T13:57:12.307 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local cluster_instance=cluster1 2023-08-26T13:57:12.307 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ grep -i state 2023-08-26T13:57:12.307 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ shift 2023-08-26T13:57:12.307 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local cluster=cluster1 2023-08-26T13:57:12.307 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local instance=cluster1 2023-08-26T13:57:12.308 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local loop_instance 2023-08-26T13:57:12.308 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-08-26T13:57:12.308 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ sleep 0 2023-08-26T13:57:12.308 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ '[' cluster1 '!=' cluster1 ']' 2023-08-26T13:57:12.309 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ seq 0 3 2023-08-26T13:57:12.310 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-08-26T13:57:12.311 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_34 2023-08-26T13:57:12.311 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local cluster=cluster1:0 2023-08-26T13:57:12.311 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ shift 2023-08-26T13:57:12.311 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local instance 2023-08-26T13:57:12.311 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ set_cluster_instance cluster1:0 cluster instance 2023-08-26T13:57:12.312 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local val=cluster1:0 2023-08-26T13:57:12.312 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local cluster_var_name=cluster 2023-08-26T13:57:12.312 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local instance_var_name=instance 2023-08-26T13:57:12.312 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ cluster=cluster1 2023-08-26T13:57:12.312 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ instance=0 2023-08-26T13:57:12.313 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ '[' 0 = cluster1:0 ']' 2023-08-26T13:57:12.313 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ eval cluster=cluster1 2023-08-26T13:57:12.313 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ cluster=cluster1 2023-08-26T13:57:12.313 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ eval instance=0 2023-08-26T13:57:12.313 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ instance=0 2023-08-26T13:57:12.314 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ daemon_asok_file cluster1:0 cluster1 2023-08-26T13:57:12.314 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ local local_cluster=cluster1:0 2023-08-26T13:57:12.314 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ local cluster=cluster1 2023-08-26T13:57:12.314 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ local instance 2023-08-26T13:57:12.314 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2023-08-26T13:57:12.314 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ local val=cluster1:0 2023-08-26T13:57:12.315 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ local cluster_var_name=local_cluster 2023-08-26T13:57:12.316 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ local instance_var_name=instance 2023-08-26T13:57:12.316 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ cluster=cluster1 2023-08-26T13:57:12.316 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ instance=0 2023-08-26T13:57:12.316 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ '[' 0 = cluster1:0 ']' 2023-08-26T13:57:12.317 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ eval local_cluster=cluster1 2023-08-26T13:57:12.317 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++++ local_cluster=cluster1 2023-08-26T13:57:12.317 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ eval instance=0 2023-08-26T13:57:12.317 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++++ instance=0 2023-08-26T13:57:12.317 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2023-08-26T13:57:12.330 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-08-26T13:57:12.331 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-08-26T13:57:12.331 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-08-26T13:57:12.331 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_34 2023-08-26T13:57:12.439 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:Invalid command: unused arguments: ['mirror/image_34'] 2023-08-26T13:57:12.440 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:rbd mirror status : get status for rbd mirror 2023-08-26T13:57:12.440 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:admin_socket: invalid command 2023-08-26T13:57:12.443 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-08-26T13:57:12.443 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_34 2023-08-26T13:57:12.443 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local cluster=cluster1:1 2023-08-26T13:57:12.443 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ shift 2023-08-26T13:57:12.443 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local instance 2023-08-26T13:57:12.444 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ set_cluster_instance cluster1:1 cluster instance 2023-08-26T13:57:12.444 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local val=cluster1:1 2023-08-26T13:57:12.444 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local cluster_var_name=cluster 2023-08-26T13:57:12.444 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local instance_var_name=instance 2023-08-26T13:57:12.444 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ cluster=cluster1 2023-08-26T13:57:12.445 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ instance=1 2023-08-26T13:57:12.445 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ '[' 1 = cluster1:1 ']' 2023-08-26T13:57:12.445 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ eval cluster=cluster1 2023-08-26T13:57:12.445 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ cluster=cluster1 2023-08-26T13:57:12.445 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ eval instance=1 2023-08-26T13:57:12.446 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ instance=1 2023-08-26T13:57:12.446 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ daemon_asok_file cluster1:1 cluster1 2023-08-26T13:57:12.446 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ local local_cluster=cluster1:1 2023-08-26T13:57:12.446 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ local cluster=cluster1 2023-08-26T13:57:12.446 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ local instance 2023-08-26T13:57:12.447 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2023-08-26T13:57:12.447 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ local val=cluster1:1 2023-08-26T13:57:12.447 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ local cluster_var_name=local_cluster 2023-08-26T13:57:12.447 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ local instance_var_name=instance 2023-08-26T13:57:12.447 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ cluster=cluster1 2023-08-26T13:57:12.448 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ instance=1 2023-08-26T13:57:12.448 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ '[' 1 = cluster1:1 ']' 2023-08-26T13:57:12.448 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ eval local_cluster=cluster1 2023-08-26T13:57:12.448 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++++ local_cluster=cluster1 2023-08-26T13:57:12.449 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ eval instance=1 2023-08-26T13:57:12.449 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++++ instance=1 2023-08-26T13:57:12.449 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2023-08-26T13:57:12.461 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-08-26T13:57:12.462 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-08-26T13:57:12.462 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-08-26T13:57:12.463 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_34 2023-08-26T13:57:12.576 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ return 0 2023-08-26T13:57:12.577 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ status_result=' "state": "Replaying"' 2023-08-26T13:57:12.577 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ echo ' "state": "Replaying"' 2023-08-26T13:57:12.578 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ grep -i Replaying 2023-08-26T13:57:12.578 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: "state": "Replaying" 2023-08-26T13:57:12.582 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ current_state=started 2023-08-26T13:57:12.582 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ test started = started 2023-08-26T13:57:12.582 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ return 0 2023-08-26T13:57:12.582 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_34 2023-08-26T13:57:12.583 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local local_cluster=cluster1 2023-08-26T13:57:12.583 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster2 2023-08-26T13:57:12.583 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T13:57:12.583 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=image_34 2023-08-26T13:57:12.583 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ '[' snapshot = journal ']' 2023-08-26T13:57:12.584 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ '[' snapshot = snapshot ']' 2023-08-26T13:57:12.584 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_34 2023-08-26T13:57:12.584 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local local_cluster=cluster1 2023-08-26T13:57:12.584 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster2 2023-08-26T13:57:12.584 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T13:57:12.585 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=image_34 2023-08-26T13:57:12.585 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ mkfname cluster2-mirror-image_34.status 2023-08-26T13:57:12.585 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ echo cluster2-mirror-image_34.status 2023-08-26T13:57:12.585 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ sed -e 's|[/ ]|_|g' 2023-08-26T13:57:12.585 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local status_log=/tmp/tmp.TfDn6BcQ3N/cluster2-mirror-image_34.status 2023-08-26T13:57:12.586 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ mkfname cluster1-mirror-image_34.status 2023-08-26T13:57:12.586 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ echo cluster1-mirror-image_34.status 2023-08-26T13:57:12.586 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ sed -e 's|[/ ]|_|g' 2023-08-26T13:57:12.586 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local local_status_log=/tmp/tmp.TfDn6BcQ3N/cluster1-mirror-image_34.status 2023-08-26T13:57:12.586 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ mirror_image_snapshot cluster2 mirror image_34 2023-08-26T13:57:12.586 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster2 2023-08-26T13:57:12.587 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T13:57:12.587 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=image_34 2023-08-26T13:57:12.587 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_34 2023-08-26T13:57:13.318 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:Snapshot ID: 185 2023-08-26T13:57:13.327 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_34 /tmp/tmp.TfDn6BcQ3N/cluster2-mirror-image_34.status 2023-08-26T13:57:13.327 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster2 2023-08-26T13:57:13.327 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T13:57:13.327 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=image_34 2023-08-26T13:57:13.328 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local log=/tmp/tmp.TfDn6BcQ3N/cluster2-mirror-image_34.status 2023-08-26T13:57:13.328 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_34 --format xml 2023-08-26T13:57:13.328 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-08-26T13:57:13.372 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ xmlstarlet sel -t -v //snapshot/id 2023-08-26T13:57:13.374 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local snapshot_id=185 2023-08-26T13:57:13.374 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ true 2023-08-26T13:57:13.374 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-08-26T13:57:13.374 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ sleep 0.2 2023-08-26T13:57:13.575 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_34 /tmp/tmp.TfDn6BcQ3N/cluster1-mirror-image_34.status 2023-08-26T13:57:13.575 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster1 2023-08-26T13:57:13.576 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T13:57:13.576 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=image_34 2023-08-26T13:57:13.576 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local log=/tmp/tmp.TfDn6BcQ3N/cluster1-mirror-image_34.status 2023-08-26T13:57:13.576 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_34 --format xml 2023-08-26T13:57:13.576 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-08-26T13:57:13.627 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ true 2023-08-26T13:57:13.628 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-08-26T13:57:13.629 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:-:1.1: Document is empty 2023-08-26T13:57:13.629 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr: 2023-08-26T13:57:13.629 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:^ 2023-08-26T13:57:13.630 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local primary_snapshot_id= 2023-08-26T13:57:13.630 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ test 185 = '' 2023-08-26T13:57:13.630 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-08-26T13:57:13.630 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ sleep 0.4 2023-08-26T13:57:14.031 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_34 /tmp/tmp.TfDn6BcQ3N/cluster1-mirror-image_34.status 2023-08-26T13:57:14.031 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster1 2023-08-26T13:57:14.031 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T13:57:14.032 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=image_34 2023-08-26T13:57:14.032 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local log=/tmp/tmp.TfDn6BcQ3N/cluster1-mirror-image_34.status 2023-08-26T13:57:14.032 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_34 --format xml 2023-08-26T13:57:14.032 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-08-26T13:57:14.080 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ true 2023-08-26T13:57:14.080 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-08-26T13:57:14.081 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:-:1.1: Document is empty 2023-08-26T13:57:14.081 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr: 2023-08-26T13:57:14.082 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:^ 2023-08-26T13:57:14.082 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local primary_snapshot_id= 2023-08-26T13:57:14.083 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ test 185 = '' 2023-08-26T13:57:14.083 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-08-26T13:57:14.083 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ sleep 0.8 2023-08-26T13:57:14.883 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_34 /tmp/tmp.TfDn6BcQ3N/cluster1-mirror-image_34.status 2023-08-26T13:57:14.883 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster1 2023-08-26T13:57:14.884 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T13:57:14.884 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=image_34 2023-08-26T13:57:14.884 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local log=/tmp/tmp.TfDn6BcQ3N/cluster1-mirror-image_34.status 2023-08-26T13:57:14.884 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_34 --format xml 2023-08-26T13:57:14.884 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-08-26T13:57:14.930 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-08-26T13:57:14.932 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local primary_snapshot_id=185 2023-08-26T13:57:14.932 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ test 185 = 185 2023-08-26T13:57:14.933 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ return 0 2023-08-26T13:57:14.933 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ wait_for_snap_present cluster1 mirror image_34 snap 2023-08-26T13:57:14.933 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster1 2023-08-26T13:57:14.933 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T13:57:14.933 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=image_34 2023-08-26T13:57:14.933 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local snap_name=snap 2023-08-26T13:57:14.934 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local s 2023-08-26T13:57:14.934 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2023-08-26T13:57:14.934 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ sleep 1 2023-08-26T13:57:15.933 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rbd --cluster cluster1 info mirror/image_34@snap 2023-08-26T13:57:15.978 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:rbd image 'image_34': 2023-08-26T13:57:15.978 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: size 128 MiB in 32 objects 2023-08-26T13:57:15.979 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: order 22 (4 MiB objects) 2023-08-26T13:57:15.979 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: snapshot_count: 2 2023-08-26T13:57:15.979 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: id: 122b44e6008b 2023-08-26T13:57:15.979 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: block_name_prefix: rbd_data.122b44e6008b 2023-08-26T13:57:15.980 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: format: 2 2023-08-26T13:57:15.980 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: features: layering, exclusive-lock, non-primary 2023-08-26T13:57:15.980 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: op_features: 2023-08-26T13:57:15.980 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: flags: 2023-08-26T13:57:15.980 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: create_timestamp: Sat Aug 26 13:48:13 2023 2023-08-26T13:57:15.981 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: access_timestamp: Sat Aug 26 13:48:13 2023 2023-08-26T13:57:15.981 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: modify_timestamp: Sat Aug 26 13:48:13 2023 2023-08-26T13:57:15.981 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: protected: False 2023-08-26T13:57:15.981 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: mirroring state: enabled 2023-08-26T13:57:15.981 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: mirroring mode: snapshot 2023-08-26T13:57:15.982 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: mirroring global id: ee57f6e1-9b01-446a-9317-2f9969ecc132 2023-08-26T13:57:15.982 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: mirroring primary: false 2023-08-26T13:57:15.983 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ return 0 2023-08-26T13:57:15.983 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ compare_image_snaps mirror image_34 snap 2023-08-26T13:57:15.984 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T13:57:15.984 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=image_34 2023-08-26T13:57:15.984 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local snap_name=snap 2023-08-26T13:57:15.984 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local ret=0 2023-08-26T13:57:15.984 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local rmt_export=/tmp/tmp.TfDn6BcQ3N/cluster2-mirror-image_34.export 2023-08-26T13:57:15.985 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local loc_export=/tmp/tmp.TfDn6BcQ3N/cluster1-mirror-image_34.export 2023-08-26T13:57:15.985 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rm -f /tmp/tmp.TfDn6BcQ3N/cluster2-mirror-image_34.export /tmp/tmp.TfDn6BcQ3N/cluster1-mirror-image_34.export 2023-08-26T13:57:15.985 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rbd --cluster cluster2 -p mirror export image_34@snap /tmp/tmp.TfDn6BcQ3N/cluster2-mirror-image_34.export 2023-08-26T13:57:16.155 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-08-26T13:57:16.161 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rbd --cluster cluster1 -p mirror export image_34@snap /tmp/tmp.TfDn6BcQ3N/cluster1-mirror-image_34.export 2023-08-26T13:57:16.384 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-08-26T13:57:16.391 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ cmp /tmp/tmp.TfDn6BcQ3N/cluster2-mirror-image_34.export /tmp/tmp.TfDn6BcQ3N/cluster1-mirror-image_34.export 2023-08-26T13:57:16.447 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rm -f /tmp/tmp.TfDn6BcQ3N/cluster2-mirror-image_34.export /tmp/tmp.TfDn6BcQ3N/cluster1-mirror-image_34.export 2023-08-26T13:57:16.484 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ return 0 2023-08-26T13:57:16.484 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-08-26T13:57:16.485 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ image=image_35 2023-08-26T13:57:16.485 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ create_snap cluster2 mirror image_35 snap 2023-08-26T13:57:16.485 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster2 2023-08-26T13:57:16.485 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T13:57:16.485 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=image_35 2023-08-26T13:57:16.485 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local snap_name=snap 2023-08-26T13:57:16.486 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rbd --cluster cluster2 -p mirror snap create image_35@snap --debug-rbd=20 --debug-journaler=20 2023-08-26T13:57:18.330 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ wait_for_image_replay_started cluster1 mirror image_35 2023-08-26T13:57:18.331 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster1 2023-08-26T13:57:18.332 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T13:57:18.332 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=image_35 2023-08-26T13:57:18.332 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ wait_for_image_replay_state cluster1 mirror image_35 started 2023-08-26T13:57:18.332 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster1 2023-08-26T13:57:18.332 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T13:57:18.332 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=image_35 2023-08-26T13:57:18.333 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local state=started 2023-08-26T13:57:18.333 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local s 2023-08-26T13:57:18.333 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-08-26T13:57:18.333 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ sleep 1 2023-08-26T13:57:19.331 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ test_image_replay_state cluster1 mirror image_35 started 2023-08-26T13:57:19.332 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster1 2023-08-26T13:57:19.332 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T13:57:19.332 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=image_35 2023-08-26T13:57:19.332 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local test_state=started 2023-08-26T13:57:19.332 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local status_result 2023-08-26T13:57:19.333 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local current_state=stopped 2023-08-26T13:57:19.334 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_35 2023-08-26T13:57:19.334 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local cluster_instance=cluster1 2023-08-26T13:57:19.334 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ shift 2023-08-26T13:57:19.334 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ grep -i state 2023-08-26T13:57:19.335 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local cluster=cluster1 2023-08-26T13:57:19.335 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local instance=cluster1 2023-08-26T13:57:19.335 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local loop_instance 2023-08-26T13:57:19.335 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-08-26T13:57:19.335 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ sleep 0 2023-08-26T13:57:19.335 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ '[' cluster1 '!=' cluster1 ']' 2023-08-26T13:57:19.336 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ seq 0 3 2023-08-26T13:57:19.338 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-08-26T13:57:19.339 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_35 2023-08-26T13:57:19.339 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local cluster=cluster1:0 2023-08-26T13:57:19.339 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ shift 2023-08-26T13:57:19.339 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local instance 2023-08-26T13:57:19.339 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ set_cluster_instance cluster1:0 cluster instance 2023-08-26T13:57:19.340 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local val=cluster1:0 2023-08-26T13:57:19.340 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local cluster_var_name=cluster 2023-08-26T13:57:19.340 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local instance_var_name=instance 2023-08-26T13:57:19.340 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ cluster=cluster1 2023-08-26T13:57:19.340 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ instance=0 2023-08-26T13:57:19.340 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ '[' 0 = cluster1:0 ']' 2023-08-26T13:57:19.341 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ eval cluster=cluster1 2023-08-26T13:57:19.341 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ cluster=cluster1 2023-08-26T13:57:19.341 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ eval instance=0 2023-08-26T13:57:19.341 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ instance=0 2023-08-26T13:57:19.341 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ daemon_asok_file cluster1:0 cluster1 2023-08-26T13:57:19.342 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ local local_cluster=cluster1:0 2023-08-26T13:57:19.342 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ local cluster=cluster1 2023-08-26T13:57:19.342 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ local instance 2023-08-26T13:57:19.342 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2023-08-26T13:57:19.342 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ local val=cluster1:0 2023-08-26T13:57:19.342 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ local cluster_var_name=local_cluster 2023-08-26T13:57:19.343 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ local instance_var_name=instance 2023-08-26T13:57:19.343 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ cluster=cluster1 2023-08-26T13:57:19.343 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ instance=0 2023-08-26T13:57:19.343 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ '[' 0 = cluster1:0 ']' 2023-08-26T13:57:19.343 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ eval local_cluster=cluster1 2023-08-26T13:57:19.344 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++++ local_cluster=cluster1 2023-08-26T13:57:19.344 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ eval instance=0 2023-08-26T13:57:19.344 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++++ instance=0 2023-08-26T13:57:19.344 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2023-08-26T13:57:19.358 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-08-26T13:57:19.358 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-08-26T13:57:19.358 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-08-26T13:57:19.359 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_35 2023-08-26T13:57:19.475 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:Invalid command: unused arguments: ['mirror/image_35'] 2023-08-26T13:57:19.475 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:rbd mirror status : get status for rbd mirror 2023-08-26T13:57:19.476 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:admin_socket: invalid command 2023-08-26T13:57:19.478 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-08-26T13:57:19.478 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_35 2023-08-26T13:57:19.479 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local cluster=cluster1:1 2023-08-26T13:57:19.479 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ shift 2023-08-26T13:57:19.479 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local instance 2023-08-26T13:57:19.479 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ set_cluster_instance cluster1:1 cluster instance 2023-08-26T13:57:19.479 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local val=cluster1:1 2023-08-26T13:57:19.480 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local cluster_var_name=cluster 2023-08-26T13:57:19.480 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local instance_var_name=instance 2023-08-26T13:57:19.480 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ cluster=cluster1 2023-08-26T13:57:19.480 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ instance=1 2023-08-26T13:57:19.480 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ '[' 1 = cluster1:1 ']' 2023-08-26T13:57:19.480 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ eval cluster=cluster1 2023-08-26T13:57:19.481 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ cluster=cluster1 2023-08-26T13:57:19.481 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ eval instance=1 2023-08-26T13:57:19.481 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ instance=1 2023-08-26T13:57:19.481 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ daemon_asok_file cluster1:1 cluster1 2023-08-26T13:57:19.481 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ local local_cluster=cluster1:1 2023-08-26T13:57:19.481 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ local cluster=cluster1 2023-08-26T13:57:19.482 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ local instance 2023-08-26T13:57:19.482 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2023-08-26T13:57:19.482 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ local val=cluster1:1 2023-08-26T13:57:19.482 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ local cluster_var_name=local_cluster 2023-08-26T13:57:19.482 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ local instance_var_name=instance 2023-08-26T13:57:19.482 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ cluster=cluster1 2023-08-26T13:57:19.483 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ instance=1 2023-08-26T13:57:19.483 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ '[' 1 = cluster1:1 ']' 2023-08-26T13:57:19.483 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ eval local_cluster=cluster1 2023-08-26T13:57:19.483 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++++ local_cluster=cluster1 2023-08-26T13:57:19.483 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ eval instance=1 2023-08-26T13:57:19.484 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++++ instance=1 2023-08-26T13:57:19.484 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2023-08-26T13:57:19.497 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-08-26T13:57:19.497 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-08-26T13:57:19.497 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-08-26T13:57:19.497 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_35 2023-08-26T13:57:19.604 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:Invalid command: unused arguments: ['mirror/image_35'] 2023-08-26T13:57:19.605 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:rbd mirror status : get status for rbd mirror 2023-08-26T13:57:19.605 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:admin_socket: invalid command 2023-08-26T13:57:19.607 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-08-26T13:57:19.608 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/image_35 2023-08-26T13:57:19.608 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local cluster=cluster1:2 2023-08-26T13:57:19.608 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ shift 2023-08-26T13:57:19.608 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local instance 2023-08-26T13:57:19.608 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ set_cluster_instance cluster1:2 cluster instance 2023-08-26T13:57:19.609 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local val=cluster1:2 2023-08-26T13:57:19.609 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local cluster_var_name=cluster 2023-08-26T13:57:19.609 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local instance_var_name=instance 2023-08-26T13:57:19.609 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ cluster=cluster1 2023-08-26T13:57:19.609 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ instance=2 2023-08-26T13:57:19.610 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ '[' 2 = cluster1:2 ']' 2023-08-26T13:57:19.610 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ eval cluster=cluster1 2023-08-26T13:57:19.610 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ cluster=cluster1 2023-08-26T13:57:19.610 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ eval instance=2 2023-08-26T13:57:19.610 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ instance=2 2023-08-26T13:57:19.611 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ daemon_asok_file cluster1:2 cluster1 2023-08-26T13:57:19.611 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ local local_cluster=cluster1:2 2023-08-26T13:57:19.611 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ local cluster=cluster1 2023-08-26T13:57:19.611 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ local instance 2023-08-26T13:57:19.611 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2023-08-26T13:57:19.611 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ local val=cluster1:2 2023-08-26T13:57:19.612 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ local cluster_var_name=local_cluster 2023-08-26T13:57:19.612 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ local instance_var_name=instance 2023-08-26T13:57:19.612 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ cluster=cluster1 2023-08-26T13:57:19.612 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ instance=2 2023-08-26T13:57:19.612 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ '[' 2 = cluster1:2 ']' 2023-08-26T13:57:19.613 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ eval local_cluster=cluster1 2023-08-26T13:57:19.613 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++++ local_cluster=cluster1 2023-08-26T13:57:19.613 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ eval instance=2 2023-08-26T13:57:19.613 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++++ instance=2 2023-08-26T13:57:19.613 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2023-08-26T13:57:19.626 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-08-26T13:57:19.626 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-08-26T13:57:19.627 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-08-26T13:57:19.627 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/image_35 2023-08-26T13:57:19.742 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ return 0 2023-08-26T13:57:19.743 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ status_result=' "state": "Replaying"' 2023-08-26T13:57:19.743 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ echo ' "state": "Replaying"' 2023-08-26T13:57:19.743 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ grep -i Replaying 2023-08-26T13:57:19.744 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: "state": "Replaying" 2023-08-26T13:57:19.746 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ current_state=started 2023-08-26T13:57:19.746 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ test started = started 2023-08-26T13:57:19.746 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ return 0 2023-08-26T13:57:19.747 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_35 2023-08-26T13:57:19.747 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local local_cluster=cluster1 2023-08-26T13:57:19.747 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster2 2023-08-26T13:57:19.747 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T13:57:19.747 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=image_35 2023-08-26T13:57:19.748 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ '[' snapshot = journal ']' 2023-08-26T13:57:19.748 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ '[' snapshot = snapshot ']' 2023-08-26T13:57:19.748 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_35 2023-08-26T13:57:19.748 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local local_cluster=cluster1 2023-08-26T13:57:19.748 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster2 2023-08-26T13:57:19.749 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T13:57:19.749 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=image_35 2023-08-26T13:57:19.749 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ mkfname cluster2-mirror-image_35.status 2023-08-26T13:57:19.749 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ echo cluster2-mirror-image_35.status 2023-08-26T13:57:19.749 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ sed -e 's|[/ ]|_|g' 2023-08-26T13:57:19.750 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local status_log=/tmp/tmp.TfDn6BcQ3N/cluster2-mirror-image_35.status 2023-08-26T13:57:19.750 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ mkfname cluster1-mirror-image_35.status 2023-08-26T13:57:19.750 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ echo cluster1-mirror-image_35.status 2023-08-26T13:57:19.750 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ sed -e 's|[/ ]|_|g' 2023-08-26T13:57:19.751 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local local_status_log=/tmp/tmp.TfDn6BcQ3N/cluster1-mirror-image_35.status 2023-08-26T13:57:19.751 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ mirror_image_snapshot cluster2 mirror image_35 2023-08-26T13:57:19.752 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster2 2023-08-26T13:57:19.752 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T13:57:19.752 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=image_35 2023-08-26T13:57:19.752 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_35 2023-08-26T13:57:20.333 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:Snapshot ID: 187 2023-08-26T13:57:20.342 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_35 /tmp/tmp.TfDn6BcQ3N/cluster2-mirror-image_35.status 2023-08-26T13:57:20.342 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster2 2023-08-26T13:57:20.342 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T13:57:20.342 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=image_35 2023-08-26T13:57:20.343 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local log=/tmp/tmp.TfDn6BcQ3N/cluster2-mirror-image_35.status 2023-08-26T13:57:20.343 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_35 --format xml 2023-08-26T13:57:20.343 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-08-26T13:57:20.385 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ xmlstarlet sel -t -v //snapshot/id 2023-08-26T13:57:20.387 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local snapshot_id=187 2023-08-26T13:57:20.388 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ true 2023-08-26T13:57:20.388 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-08-26T13:57:20.388 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ sleep 0.2 2023-08-26T13:57:20.589 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_35 /tmp/tmp.TfDn6BcQ3N/cluster1-mirror-image_35.status 2023-08-26T13:57:20.589 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster1 2023-08-26T13:57:20.589 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T13:57:20.589 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=image_35 2023-08-26T13:57:20.590 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local log=/tmp/tmp.TfDn6BcQ3N/cluster1-mirror-image_35.status 2023-08-26T13:57:20.590 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_35 --format xml 2023-08-26T13:57:20.590 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-08-26T13:57:20.636 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-08-26T13:57:20.638 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local primary_snapshot_id=102 2023-08-26T13:57:20.638 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ test 187 = 102 2023-08-26T13:57:20.638 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-08-26T13:57:20.638 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ sleep 0.4 2023-08-26T13:57:21.039 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_35 /tmp/tmp.TfDn6BcQ3N/cluster1-mirror-image_35.status 2023-08-26T13:57:21.040 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster1 2023-08-26T13:57:21.040 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T13:57:21.040 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=image_35 2023-08-26T13:57:21.040 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local log=/tmp/tmp.TfDn6BcQ3N/cluster1-mirror-image_35.status 2023-08-26T13:57:21.040 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_35 --format xml 2023-08-26T13:57:21.041 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-08-26T13:57:21.085 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-08-26T13:57:21.087 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local primary_snapshot_id=102 2023-08-26T13:57:21.088 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ test 187 = 102 2023-08-26T13:57:21.088 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-08-26T13:57:21.088 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ sleep 0.8 2023-08-26T13:57:21.889 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_35 /tmp/tmp.TfDn6BcQ3N/cluster1-mirror-image_35.status 2023-08-26T13:57:21.889 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster1 2023-08-26T13:57:21.890 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T13:57:21.890 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=image_35 2023-08-26T13:57:21.890 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local log=/tmp/tmp.TfDn6BcQ3N/cluster1-mirror-image_35.status 2023-08-26T13:57:21.890 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_35 --format xml 2023-08-26T13:57:21.890 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-08-26T13:57:21.942 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ true 2023-08-26T13:57:21.942 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-08-26T13:57:21.943 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:-:1.1: Document is empty 2023-08-26T13:57:21.943 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr: 2023-08-26T13:57:21.944 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:^ 2023-08-26T13:57:21.944 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local primary_snapshot_id= 2023-08-26T13:57:21.944 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ test 187 = '' 2023-08-26T13:57:21.945 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-08-26T13:57:21.945 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ sleep 1.6 2023-08-26T13:57:23.545 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_35 /tmp/tmp.TfDn6BcQ3N/cluster1-mirror-image_35.status 2023-08-26T13:57:23.546 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster1 2023-08-26T13:57:23.546 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T13:57:23.546 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=image_35 2023-08-26T13:57:23.546 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local log=/tmp/tmp.TfDn6BcQ3N/cluster1-mirror-image_35.status 2023-08-26T13:57:23.546 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_35 --format xml 2023-08-26T13:57:23.547 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-08-26T13:57:23.606 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-08-26T13:57:23.609 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local primary_snapshot_id=187 2023-08-26T13:57:23.609 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ test 187 = 187 2023-08-26T13:57:23.609 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ return 0 2023-08-26T13:57:23.609 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ wait_for_snap_present cluster1 mirror image_35 snap 2023-08-26T13:57:23.610 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster1 2023-08-26T13:57:23.610 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T13:57:23.610 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=image_35 2023-08-26T13:57:23.610 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local snap_name=snap 2023-08-26T13:57:23.610 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local s 2023-08-26T13:57:23.611 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2023-08-26T13:57:23.611 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ sleep 1 2023-08-26T13:57:24.610 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rbd --cluster cluster1 info mirror/image_35@snap 2023-08-26T13:57:24.655 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:rbd image 'image_35': 2023-08-26T13:57:24.655 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: size 128 MiB in 32 objects 2023-08-26T13:57:24.655 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: order 22 (4 MiB objects) 2023-08-26T13:57:24.655 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: snapshot_count: 2 2023-08-26T13:57:24.655 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: id: 12c072b6ba87 2023-08-26T13:57:24.656 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: block_name_prefix: rbd_data.12c072b6ba87 2023-08-26T13:57:24.656 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: format: 2 2023-08-26T13:57:24.656 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: features: layering, exclusive-lock, non-primary 2023-08-26T13:57:24.656 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: op_features: 2023-08-26T13:57:24.657 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: flags: 2023-08-26T13:57:24.657 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: create_timestamp: Sat Aug 26 13:48:22 2023 2023-08-26T13:57:24.657 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: access_timestamp: Sat Aug 26 13:48:22 2023 2023-08-26T13:57:24.657 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: modify_timestamp: Sat Aug 26 13:48:22 2023 2023-08-26T13:57:24.657 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: protected: False 2023-08-26T13:57:24.658 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: mirroring state: enabled 2023-08-26T13:57:24.658 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: mirroring mode: snapshot 2023-08-26T13:57:24.658 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: mirroring global id: 92367d6b-5947-4538-a25d-c7bdd622153a 2023-08-26T13:57:24.658 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: mirroring primary: false 2023-08-26T13:57:24.660 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ return 0 2023-08-26T13:57:24.660 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ compare_image_snaps mirror image_35 snap 2023-08-26T13:57:24.660 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T13:57:24.660 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=image_35 2023-08-26T13:57:24.661 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local snap_name=snap 2023-08-26T13:57:24.661 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local ret=0 2023-08-26T13:57:24.661 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local rmt_export=/tmp/tmp.TfDn6BcQ3N/cluster2-mirror-image_35.export 2023-08-26T13:57:24.661 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local loc_export=/tmp/tmp.TfDn6BcQ3N/cluster1-mirror-image_35.export 2023-08-26T13:57:24.661 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rm -f /tmp/tmp.TfDn6BcQ3N/cluster2-mirror-image_35.export /tmp/tmp.TfDn6BcQ3N/cluster1-mirror-image_35.export 2023-08-26T13:57:24.662 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rbd --cluster cluster2 -p mirror export image_35@snap /tmp/tmp.TfDn6BcQ3N/cluster2-mirror-image_35.export 2023-08-26T13:57:24.837 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-08-26T13:57:24.843 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rbd --cluster cluster1 -p mirror export image_35@snap /tmp/tmp.TfDn6BcQ3N/cluster1-mirror-image_35.export 2023-08-26T13:57:25.082 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-08-26T13:57:25.090 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ cmp /tmp/tmp.TfDn6BcQ3N/cluster2-mirror-image_35.export /tmp/tmp.TfDn6BcQ3N/cluster1-mirror-image_35.export 2023-08-26T13:57:25.144 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rm -f /tmp/tmp.TfDn6BcQ3N/cluster2-mirror-image_35.export /tmp/tmp.TfDn6BcQ3N/cluster1-mirror-image_35.export 2023-08-26T13:57:25.181 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ return 0 2023-08-26T13:57:25.182 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-08-26T13:57:25.182 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ image=image_36 2023-08-26T13:57:25.182 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ create_snap cluster2 mirror image_36 snap 2023-08-26T13:57:25.182 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster2 2023-08-26T13:57:25.183 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T13:57:25.183 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=image_36 2023-08-26T13:57:25.183 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local snap_name=snap 2023-08-26T13:57:25.183 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rbd --cluster cluster2 -p mirror snap create image_36@snap --debug-rbd=20 --debug-journaler=20 2023-08-26T13:57:27.205 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ wait_for_image_replay_started cluster1 mirror image_36 2023-08-26T13:57:27.205 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster1 2023-08-26T13:57:27.205 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T13:57:27.206 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=image_36 2023-08-26T13:57:27.206 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ wait_for_image_replay_state cluster1 mirror image_36 started 2023-08-26T13:57:27.206 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster1 2023-08-26T13:57:27.206 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T13:57:27.206 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=image_36 2023-08-26T13:57:27.206 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local state=started 2023-08-26T13:57:27.207 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local s 2023-08-26T13:57:27.207 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-08-26T13:57:27.207 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ sleep 1 2023-08-26T13:57:28.206 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ test_image_replay_state cluster1 mirror image_36 started 2023-08-26T13:57:28.206 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster1 2023-08-26T13:57:28.207 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T13:57:28.207 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=image_36 2023-08-26T13:57:28.207 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local test_state=started 2023-08-26T13:57:28.207 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local status_result 2023-08-26T13:57:28.208 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local current_state=stopped 2023-08-26T13:57:28.211 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_36 2023-08-26T13:57:28.212 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ grep -i state 2023-08-26T13:57:28.212 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local cluster_instance=cluster1 2023-08-26T13:57:28.212 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ shift 2023-08-26T13:57:28.213 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local cluster=cluster1 2023-08-26T13:57:28.213 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local instance=cluster1 2023-08-26T13:57:28.214 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local loop_instance 2023-08-26T13:57:28.214 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-08-26T13:57:28.214 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ sleep 0 2023-08-26T13:57:28.214 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ '[' cluster1 '!=' cluster1 ']' 2023-08-26T13:57:28.215 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ seq 0 3 2023-08-26T13:57:28.215 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-08-26T13:57:28.215 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_36 2023-08-26T13:57:28.215 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local cluster=cluster1:0 2023-08-26T13:57:28.215 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ shift 2023-08-26T13:57:28.216 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local instance 2023-08-26T13:57:28.216 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ set_cluster_instance cluster1:0 cluster instance 2023-08-26T13:57:28.216 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local val=cluster1:0 2023-08-26T13:57:28.217 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local cluster_var_name=cluster 2023-08-26T13:57:28.217 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local instance_var_name=instance 2023-08-26T13:57:28.217 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ cluster=cluster1 2023-08-26T13:57:28.217 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ instance=0 2023-08-26T13:57:28.217 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ '[' 0 = cluster1:0 ']' 2023-08-26T13:57:28.218 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ eval cluster=cluster1 2023-08-26T13:57:28.218 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ cluster=cluster1 2023-08-26T13:57:28.218 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ eval instance=0 2023-08-26T13:57:28.218 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ instance=0 2023-08-26T13:57:28.218 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ daemon_asok_file cluster1:0 cluster1 2023-08-26T13:57:28.219 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ local local_cluster=cluster1:0 2023-08-26T13:57:28.219 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ local cluster=cluster1 2023-08-26T13:57:28.219 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ local instance 2023-08-26T13:57:28.219 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2023-08-26T13:57:28.219 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ local val=cluster1:0 2023-08-26T13:57:28.220 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ local cluster_var_name=local_cluster 2023-08-26T13:57:28.220 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ local instance_var_name=instance 2023-08-26T13:57:28.220 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ cluster=cluster1 2023-08-26T13:57:28.220 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ instance=0 2023-08-26T13:57:28.220 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ '[' 0 = cluster1:0 ']' 2023-08-26T13:57:28.220 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ eval local_cluster=cluster1 2023-08-26T13:57:28.221 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++++ local_cluster=cluster1 2023-08-26T13:57:28.221 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ eval instance=0 2023-08-26T13:57:28.221 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++++ instance=0 2023-08-26T13:57:28.221 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2023-08-26T13:57:28.233 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-08-26T13:57:28.233 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-08-26T13:57:28.233 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-08-26T13:57:28.234 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_36 2023-08-26T13:57:28.349 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:Invalid command: unused arguments: ['mirror/image_36'] 2023-08-26T13:57:28.349 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:rbd mirror status : get status for rbd mirror 2023-08-26T13:57:28.349 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:admin_socket: invalid command 2023-08-26T13:57:28.352 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-08-26T13:57:28.352 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_36 2023-08-26T13:57:28.352 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local cluster=cluster1:1 2023-08-26T13:57:28.353 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ shift 2023-08-26T13:57:28.353 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local instance 2023-08-26T13:57:28.353 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ set_cluster_instance cluster1:1 cluster instance 2023-08-26T13:57:28.353 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local val=cluster1:1 2023-08-26T13:57:28.353 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local cluster_var_name=cluster 2023-08-26T13:57:28.353 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local instance_var_name=instance 2023-08-26T13:57:28.354 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ cluster=cluster1 2023-08-26T13:57:28.354 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ instance=1 2023-08-26T13:57:28.354 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ '[' 1 = cluster1:1 ']' 2023-08-26T13:57:28.354 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ eval cluster=cluster1 2023-08-26T13:57:28.354 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ cluster=cluster1 2023-08-26T13:57:28.355 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ eval instance=1 2023-08-26T13:57:28.355 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ instance=1 2023-08-26T13:57:28.355 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ daemon_asok_file cluster1:1 cluster1 2023-08-26T13:57:28.355 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ local local_cluster=cluster1:1 2023-08-26T13:57:28.355 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ local cluster=cluster1 2023-08-26T13:57:28.355 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ local instance 2023-08-26T13:57:28.356 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2023-08-26T13:57:28.356 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ local val=cluster1:1 2023-08-26T13:57:28.356 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ local cluster_var_name=local_cluster 2023-08-26T13:57:28.356 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ local instance_var_name=instance 2023-08-26T13:57:28.356 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ cluster=cluster1 2023-08-26T13:57:28.357 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ instance=1 2023-08-26T13:57:28.357 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ '[' 1 = cluster1:1 ']' 2023-08-26T13:57:28.357 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ eval local_cluster=cluster1 2023-08-26T13:57:28.357 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++++ local_cluster=cluster1 2023-08-26T13:57:28.357 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ eval instance=1 2023-08-26T13:57:28.358 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++++ instance=1 2023-08-26T13:57:28.358 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2023-08-26T13:57:28.371 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-08-26T13:57:28.371 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-08-26T13:57:28.371 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-08-26T13:57:28.371 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_36 2023-08-26T13:57:28.478 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:Invalid command: unused arguments: ['mirror/image_36'] 2023-08-26T13:57:28.479 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:rbd mirror status : get status for rbd mirror 2023-08-26T13:57:28.479 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:admin_socket: invalid command 2023-08-26T13:57:28.482 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-08-26T13:57:28.482 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/image_36 2023-08-26T13:57:28.482 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local cluster=cluster1:2 2023-08-26T13:57:28.482 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ shift 2023-08-26T13:57:28.483 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local instance 2023-08-26T13:57:28.483 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ set_cluster_instance cluster1:2 cluster instance 2023-08-26T13:57:28.483 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local val=cluster1:2 2023-08-26T13:57:28.483 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local cluster_var_name=cluster 2023-08-26T13:57:28.483 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local instance_var_name=instance 2023-08-26T13:57:28.484 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ cluster=cluster1 2023-08-26T13:57:28.484 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ instance=2 2023-08-26T13:57:28.484 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ '[' 2 = cluster1:2 ']' 2023-08-26T13:57:28.484 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ eval cluster=cluster1 2023-08-26T13:57:28.484 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ cluster=cluster1 2023-08-26T13:57:28.484 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ eval instance=2 2023-08-26T13:57:28.485 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ instance=2 2023-08-26T13:57:28.485 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ daemon_asok_file cluster1:2 cluster1 2023-08-26T13:57:28.485 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ local local_cluster=cluster1:2 2023-08-26T13:57:28.485 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ local cluster=cluster1 2023-08-26T13:57:28.485 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ local instance 2023-08-26T13:57:28.486 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2023-08-26T13:57:28.486 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ local val=cluster1:2 2023-08-26T13:57:28.486 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ local cluster_var_name=local_cluster 2023-08-26T13:57:28.486 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ local instance_var_name=instance 2023-08-26T13:57:28.486 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ cluster=cluster1 2023-08-26T13:57:28.486 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ instance=2 2023-08-26T13:57:28.487 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ '[' 2 = cluster1:2 ']' 2023-08-26T13:57:28.487 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ eval local_cluster=cluster1 2023-08-26T13:57:28.487 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++++ local_cluster=cluster1 2023-08-26T13:57:28.487 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ eval instance=2 2023-08-26T13:57:28.487 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++++ instance=2 2023-08-26T13:57:28.488 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2023-08-26T13:57:28.500 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-08-26T13:57:28.500 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-08-26T13:57:28.500 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-08-26T13:57:28.501 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/image_36 2023-08-26T13:57:28.614 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ return 0 2023-08-26T13:57:28.614 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ status_result=' "state": "Replaying"' 2023-08-26T13:57:28.615 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ echo ' "state": "Replaying"' 2023-08-26T13:57:28.615 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ grep -i Replaying 2023-08-26T13:57:28.615 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: "state": "Replaying" 2023-08-26T13:57:28.618 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ current_state=started 2023-08-26T13:57:28.618 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ test started = started 2023-08-26T13:57:28.618 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ return 0 2023-08-26T13:57:28.618 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_36 2023-08-26T13:57:28.618 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local local_cluster=cluster1 2023-08-26T13:57:28.619 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster2 2023-08-26T13:57:28.619 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T13:57:28.619 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=image_36 2023-08-26T13:57:28.619 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ '[' snapshot = journal ']' 2023-08-26T13:57:28.619 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ '[' snapshot = snapshot ']' 2023-08-26T13:57:28.620 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_36 2023-08-26T13:57:28.620 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local local_cluster=cluster1 2023-08-26T13:57:28.620 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster2 2023-08-26T13:57:28.620 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T13:57:28.620 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=image_36 2023-08-26T13:57:28.620 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ mkfname cluster2-mirror-image_36.status 2023-08-26T13:57:28.621 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ echo cluster2-mirror-image_36.status 2023-08-26T13:57:28.621 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ sed -e 's|[/ ]|_|g' 2023-08-26T13:57:28.621 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local status_log=/tmp/tmp.TfDn6BcQ3N/cluster2-mirror-image_36.status 2023-08-26T13:57:28.621 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ mkfname cluster1-mirror-image_36.status 2023-08-26T13:57:28.621 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ echo cluster1-mirror-image_36.status 2023-08-26T13:57:28.622 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ sed -e 's|[/ ]|_|g' 2023-08-26T13:57:28.622 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local local_status_log=/tmp/tmp.TfDn6BcQ3N/cluster1-mirror-image_36.status 2023-08-26T13:57:28.623 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ mirror_image_snapshot cluster2 mirror image_36 2023-08-26T13:57:28.623 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster2 2023-08-26T13:57:28.623 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T13:57:28.623 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=image_36 2023-08-26T13:57:28.623 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_36 2023-08-26T13:57:29.208 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:Snapshot ID: 189 2023-08-26T13:57:29.217 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_36 /tmp/tmp.TfDn6BcQ3N/cluster2-mirror-image_36.status 2023-08-26T13:57:29.217 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster2 2023-08-26T13:57:29.218 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T13:57:29.218 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=image_36 2023-08-26T13:57:29.218 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local log=/tmp/tmp.TfDn6BcQ3N/cluster2-mirror-image_36.status 2023-08-26T13:57:29.218 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_36 --format xml 2023-08-26T13:57:29.218 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-08-26T13:57:29.259 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ xmlstarlet sel -t -v //snapshot/id 2023-08-26T13:57:29.261 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local snapshot_id=189 2023-08-26T13:57:29.261 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ true 2023-08-26T13:57:29.262 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-08-26T13:57:29.262 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ sleep 0.2 2023-08-26T13:57:29.463 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_36 /tmp/tmp.TfDn6BcQ3N/cluster1-mirror-image_36.status 2023-08-26T13:57:29.463 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster1 2023-08-26T13:57:29.463 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T13:57:29.463 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=image_36 2023-08-26T13:57:29.463 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local log=/tmp/tmp.TfDn6BcQ3N/cluster1-mirror-image_36.status 2023-08-26T13:57:29.464 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_36 --format xml 2023-08-26T13:57:29.464 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-08-26T13:57:29.509 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-08-26T13:57:29.511 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local primary_snapshot_id=103 2023-08-26T13:57:29.511 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ test 189 = 103 2023-08-26T13:57:29.511 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-08-26T13:57:29.511 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ sleep 0.4 2023-08-26T13:57:29.913 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_36 /tmp/tmp.TfDn6BcQ3N/cluster1-mirror-image_36.status 2023-08-26T13:57:29.913 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster1 2023-08-26T13:57:29.913 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T13:57:29.913 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=image_36 2023-08-26T13:57:29.914 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local log=/tmp/tmp.TfDn6BcQ3N/cluster1-mirror-image_36.status 2023-08-26T13:57:29.914 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_36 --format xml 2023-08-26T13:57:29.914 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-08-26T13:57:29.959 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ true 2023-08-26T13:57:29.960 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-08-26T13:57:29.961 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:-:1.1: Document is empty 2023-08-26T13:57:29.961 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr: 2023-08-26T13:57:29.961 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:^ 2023-08-26T13:57:29.962 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local primary_snapshot_id= 2023-08-26T13:57:29.962 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ test 189 = '' 2023-08-26T13:57:29.962 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-08-26T13:57:29.963 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ sleep 0.8 2023-08-26T13:57:30.763 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_36 /tmp/tmp.TfDn6BcQ3N/cluster1-mirror-image_36.status 2023-08-26T13:57:30.763 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster1 2023-08-26T13:57:30.764 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T13:57:30.764 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=image_36 2023-08-26T13:57:30.764 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local log=/tmp/tmp.TfDn6BcQ3N/cluster1-mirror-image_36.status 2023-08-26T13:57:30.764 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_36 --format xml 2023-08-26T13:57:30.765 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-08-26T13:57:30.807 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-08-26T13:57:30.809 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local primary_snapshot_id=189 2023-08-26T13:57:30.809 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ test 189 = 189 2023-08-26T13:57:30.809 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ return 0 2023-08-26T13:57:30.809 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ wait_for_snap_present cluster1 mirror image_36 snap 2023-08-26T13:57:30.810 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster1 2023-08-26T13:57:30.810 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T13:57:30.810 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=image_36 2023-08-26T13:57:30.810 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local snap_name=snap 2023-08-26T13:57:30.810 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local s 2023-08-26T13:57:30.811 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2023-08-26T13:57:30.811 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ sleep 1 2023-08-26T13:57:31.809 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rbd --cluster cluster1 info mirror/image_36@snap 2023-08-26T13:57:31.851 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:rbd image 'image_36': 2023-08-26T13:57:31.852 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: size 128 MiB in 32 objects 2023-08-26T13:57:31.852 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: order 22 (4 MiB objects) 2023-08-26T13:57:31.852 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: snapshot_count: 2 2023-08-26T13:57:31.852 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: id: 12cffc330376 2023-08-26T13:57:31.852 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: block_name_prefix: rbd_data.12cffc330376 2023-08-26T13:57:31.853 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: format: 2 2023-08-26T13:57:31.853 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: features: layering, exclusive-lock, non-primary 2023-08-26T13:57:31.853 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: op_features: 2023-08-26T13:57:31.853 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: flags: 2023-08-26T13:57:31.853 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: create_timestamp: Sat Aug 26 13:48:36 2023 2023-08-26T13:57:31.853 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: access_timestamp: Sat Aug 26 13:48:36 2023 2023-08-26T13:57:31.854 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: modify_timestamp: Sat Aug 26 13:48:36 2023 2023-08-26T13:57:31.854 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: protected: False 2023-08-26T13:57:31.854 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: mirroring state: enabled 2023-08-26T13:57:31.854 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: mirroring mode: snapshot 2023-08-26T13:57:31.854 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: mirroring global id: 8068d21b-b929-4466-9d32-bbd96d0ca61c 2023-08-26T13:57:31.855 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: mirroring primary: false 2023-08-26T13:57:31.856 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ return 0 2023-08-26T13:57:31.856 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ compare_image_snaps mirror image_36 snap 2023-08-26T13:57:31.856 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T13:57:31.856 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=image_36 2023-08-26T13:57:31.857 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local snap_name=snap 2023-08-26T13:57:31.857 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local ret=0 2023-08-26T13:57:31.857 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local rmt_export=/tmp/tmp.TfDn6BcQ3N/cluster2-mirror-image_36.export 2023-08-26T13:57:31.857 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local loc_export=/tmp/tmp.TfDn6BcQ3N/cluster1-mirror-image_36.export 2023-08-26T13:57:31.857 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rm -f /tmp/tmp.TfDn6BcQ3N/cluster2-mirror-image_36.export /tmp/tmp.TfDn6BcQ3N/cluster1-mirror-image_36.export 2023-08-26T13:57:31.857 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rbd --cluster cluster2 -p mirror export image_36@snap /tmp/tmp.TfDn6BcQ3N/cluster2-mirror-image_36.export 2023-08-26T13:57:32.041 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-08-26T13:57:32.046 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rbd --cluster cluster1 -p mirror export image_36@snap /tmp/tmp.TfDn6BcQ3N/cluster1-mirror-image_36.export 2023-08-26T13:57:32.281 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-08-26T13:57:32.288 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ cmp /tmp/tmp.TfDn6BcQ3N/cluster2-mirror-image_36.export /tmp/tmp.TfDn6BcQ3N/cluster1-mirror-image_36.export 2023-08-26T13:57:32.343 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rm -f /tmp/tmp.TfDn6BcQ3N/cluster2-mirror-image_36.export /tmp/tmp.TfDn6BcQ3N/cluster1-mirror-image_36.export 2023-08-26T13:57:32.379 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ return 0 2023-08-26T13:57:32.380 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-08-26T13:57:32.380 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ image=image_37 2023-08-26T13:57:32.380 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ create_snap cluster2 mirror image_37 snap 2023-08-26T13:57:32.380 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster2 2023-08-26T13:57:32.381 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T13:57:32.381 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=image_37 2023-08-26T13:57:32.381 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local snap_name=snap 2023-08-26T13:57:32.381 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rbd --cluster cluster2 -p mirror snap create image_37@snap --debug-rbd=20 --debug-journaler=20 2023-08-26T13:57:33.536 INFO:tasks.rbd_mirror_thrash:kill cluster2.client.mirror.0 2023-08-26T13:57:33.537 INFO:tasks.rbd_mirror.cluster2.client.mirror.0:Sent signal 15 2023-08-26T13:57:33.537 INFO:tasks.rbd_mirror_thrash:waiting for 1 secs before reviving daemons 2023-08-26T13:57:33.538 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi124.stderr:2023-08-26T13:57:33.537+0000 7fe4b7632700 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.0 (PID: 128014) UID: 1000 2023-08-26T13:57:34.225 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ wait_for_image_replay_started cluster1 mirror image_37 2023-08-26T13:57:34.225 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster1 2023-08-26T13:57:34.225 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T13:57:34.225 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=image_37 2023-08-26T13:57:34.226 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ wait_for_image_replay_state cluster1 mirror image_37 started 2023-08-26T13:57:34.226 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster1 2023-08-26T13:57:34.226 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T13:57:34.226 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=image_37 2023-08-26T13:57:34.226 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local state=started 2023-08-26T13:57:34.226 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local s 2023-08-26T13:57:34.227 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-08-26T13:57:34.227 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ sleep 1 2023-08-26T13:57:34.538 INFO:tasks.rbd_mirror_thrash:waiting for cluster2.client.mirror.0 2023-08-26T13:57:34.538 INFO:teuthology.orchestra.run:waiting for 600 2023-08-26T13:57:34.538 INFO:tasks.rbd_mirror_thrash:reviving cluster2.client.mirror.0 2023-08-26T13:57:34.539 INFO:tasks.rbd_mirror.cluster2.client.mirror.0:Restarting daemon 2023-08-26T13:57:34.539 DEBUG:teuthology.orchestra.run.smithi124:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.0 2023-08-26T13:57:34.542 INFO:tasks.rbd_mirror.cluster2.client.mirror.0:Started 2023-08-26T13:57:34.542 INFO:tasks.rbd_mirror_thrash:waiting for 64 secs before thrashing 2023-08-26T13:57:35.227 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ test_image_replay_state cluster1 mirror image_37 started 2023-08-26T13:57:35.227 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster1 2023-08-26T13:57:35.227 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T13:57:35.228 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=image_37 2023-08-26T13:57:35.228 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local test_state=started 2023-08-26T13:57:35.228 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local status_result 2023-08-26T13:57:35.228 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local current_state=stopped 2023-08-26T13:57:35.228 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_37 2023-08-26T13:57:35.229 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ grep -i state 2023-08-26T13:57:35.229 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local cluster_instance=cluster1 2023-08-26T13:57:35.229 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ shift 2023-08-26T13:57:35.229 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local cluster=cluster1 2023-08-26T13:57:35.229 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local instance=cluster1 2023-08-26T13:57:35.230 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local loop_instance 2023-08-26T13:57:35.230 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-08-26T13:57:35.230 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ sleep 0 2023-08-26T13:57:35.232 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ '[' cluster1 '!=' cluster1 ']' 2023-08-26T13:57:35.232 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ seq 0 3 2023-08-26T13:57:35.232 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-08-26T13:57:35.233 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_37 2023-08-26T13:57:35.233 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local cluster=cluster1:0 2023-08-26T13:57:35.233 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ shift 2023-08-26T13:57:35.233 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local instance 2023-08-26T13:57:35.233 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ set_cluster_instance cluster1:0 cluster instance 2023-08-26T13:57:35.233 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local val=cluster1:0 2023-08-26T13:57:35.234 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local cluster_var_name=cluster 2023-08-26T13:57:35.234 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local instance_var_name=instance 2023-08-26T13:57:35.234 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ cluster=cluster1 2023-08-26T13:57:35.234 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ instance=0 2023-08-26T13:57:35.234 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ '[' 0 = cluster1:0 ']' 2023-08-26T13:57:35.235 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ eval cluster=cluster1 2023-08-26T13:57:35.235 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ cluster=cluster1 2023-08-26T13:57:35.237 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ eval instance=0 2023-08-26T13:57:35.237 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ instance=0 2023-08-26T13:57:35.237 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ daemon_asok_file cluster1:0 cluster1 2023-08-26T13:57:35.237 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ local local_cluster=cluster1:0 2023-08-26T13:57:35.237 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ local cluster=cluster1 2023-08-26T13:57:35.238 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ local instance 2023-08-26T13:57:35.238 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2023-08-26T13:57:35.238 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ local val=cluster1:0 2023-08-26T13:57:35.238 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ local cluster_var_name=local_cluster 2023-08-26T13:57:35.238 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ local instance_var_name=instance 2023-08-26T13:57:35.239 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ cluster=cluster1 2023-08-26T13:57:35.239 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ instance=0 2023-08-26T13:57:35.239 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ '[' 0 = cluster1:0 ']' 2023-08-26T13:57:35.239 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ eval local_cluster=cluster1 2023-08-26T13:57:35.239 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++++ local_cluster=cluster1 2023-08-26T13:57:35.240 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ eval instance=0 2023-08-26T13:57:35.240 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++++ instance=0 2023-08-26T13:57:35.240 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2023-08-26T13:57:35.249 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-08-26T13:57:35.250 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-08-26T13:57:35.250 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-08-26T13:57:35.250 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_37 2023-08-26T13:57:35.354 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:Invalid command: unused arguments: ['mirror/image_37'] 2023-08-26T13:57:35.354 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:rbd mirror status : get status for rbd mirror 2023-08-26T13:57:35.355 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:admin_socket: invalid command 2023-08-26T13:57:35.357 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-08-26T13:57:35.357 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_37 2023-08-26T13:57:35.358 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local cluster=cluster1:1 2023-08-26T13:57:35.358 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ shift 2023-08-26T13:57:35.358 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local instance 2023-08-26T13:57:35.358 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ set_cluster_instance cluster1:1 cluster instance 2023-08-26T13:57:35.358 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local val=cluster1:1 2023-08-26T13:57:35.359 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local cluster_var_name=cluster 2023-08-26T13:57:35.359 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local instance_var_name=instance 2023-08-26T13:57:35.359 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ cluster=cluster1 2023-08-26T13:57:35.359 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ instance=1 2023-08-26T13:57:35.359 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ '[' 1 = cluster1:1 ']' 2023-08-26T13:57:35.359 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ eval cluster=cluster1 2023-08-26T13:57:35.360 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ cluster=cluster1 2023-08-26T13:57:35.360 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ eval instance=1 2023-08-26T13:57:35.360 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ instance=1 2023-08-26T13:57:35.360 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ daemon_asok_file cluster1:1 cluster1 2023-08-26T13:57:35.360 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ local local_cluster=cluster1:1 2023-08-26T13:57:35.361 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ local cluster=cluster1 2023-08-26T13:57:35.361 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ local instance 2023-08-26T13:57:35.361 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2023-08-26T13:57:35.361 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ local val=cluster1:1 2023-08-26T13:57:35.361 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ local cluster_var_name=local_cluster 2023-08-26T13:57:35.361 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ local instance_var_name=instance 2023-08-26T13:57:35.362 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ cluster=cluster1 2023-08-26T13:57:35.362 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ instance=1 2023-08-26T13:57:35.362 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ '[' 1 = cluster1:1 ']' 2023-08-26T13:57:35.362 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ eval local_cluster=cluster1 2023-08-26T13:57:35.362 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++++ local_cluster=cluster1 2023-08-26T13:57:35.362 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ eval instance=1 2023-08-26T13:57:35.363 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++++ instance=1 2023-08-26T13:57:35.363 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2023-08-26T13:57:35.376 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-08-26T13:57:35.376 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-08-26T13:57:35.376 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-08-26T13:57:35.376 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_37 2023-08-26T13:57:35.484 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:Invalid command: unused arguments: ['mirror/image_37'] 2023-08-26T13:57:35.484 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:rbd mirror status : get status for rbd mirror 2023-08-26T13:57:35.484 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:admin_socket: invalid command 2023-08-26T13:57:35.487 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-08-26T13:57:35.487 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/image_37 2023-08-26T13:57:35.488 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local cluster=cluster1:2 2023-08-26T13:57:35.488 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ shift 2023-08-26T13:57:35.488 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local instance 2023-08-26T13:57:35.488 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ set_cluster_instance cluster1:2 cluster instance 2023-08-26T13:57:35.488 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local val=cluster1:2 2023-08-26T13:57:35.489 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local cluster_var_name=cluster 2023-08-26T13:57:35.489 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local instance_var_name=instance 2023-08-26T13:57:35.489 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ cluster=cluster1 2023-08-26T13:57:35.489 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ instance=2 2023-08-26T13:57:35.489 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ '[' 2 = cluster1:2 ']' 2023-08-26T13:57:35.490 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ eval cluster=cluster1 2023-08-26T13:57:35.490 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ cluster=cluster1 2023-08-26T13:57:35.490 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ eval instance=2 2023-08-26T13:57:35.490 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ instance=2 2023-08-26T13:57:35.490 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ daemon_asok_file cluster1:2 cluster1 2023-08-26T13:57:35.491 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ local local_cluster=cluster1:2 2023-08-26T13:57:35.491 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ local cluster=cluster1 2023-08-26T13:57:35.491 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ local instance 2023-08-26T13:57:35.491 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2023-08-26T13:57:35.491 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ local val=cluster1:2 2023-08-26T13:57:35.491 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ local cluster_var_name=local_cluster 2023-08-26T13:57:35.492 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ local instance_var_name=instance 2023-08-26T13:57:35.492 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ cluster=cluster1 2023-08-26T13:57:35.492 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ instance=2 2023-08-26T13:57:35.492 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ '[' 2 = cluster1:2 ']' 2023-08-26T13:57:35.492 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ eval local_cluster=cluster1 2023-08-26T13:57:35.493 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++++ local_cluster=cluster1 2023-08-26T13:57:35.493 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ eval instance=2 2023-08-26T13:57:35.493 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++++ instance=2 2023-08-26T13:57:35.493 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2023-08-26T13:57:35.507 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-08-26T13:57:35.507 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-08-26T13:57:35.507 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-08-26T13:57:35.507 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/image_37 2023-08-26T13:57:35.611 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:Invalid command: unused arguments: ['mirror/image_37'] 2023-08-26T13:57:35.611 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:rbd mirror status : get status for rbd mirror 2023-08-26T13:57:35.611 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:admin_socket: invalid command 2023-08-26T13:57:35.614 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-08-26T13:57:35.615 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ admin_daemon cluster1:3 rbd mirror status mirror/image_37 2023-08-26T13:57:35.615 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local cluster=cluster1:3 2023-08-26T13:57:35.615 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ shift 2023-08-26T13:57:35.615 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local instance 2023-08-26T13:57:35.615 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ set_cluster_instance cluster1:3 cluster instance 2023-08-26T13:57:35.615 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local val=cluster1:3 2023-08-26T13:57:35.616 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local cluster_var_name=cluster 2023-08-26T13:57:35.616 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local instance_var_name=instance 2023-08-26T13:57:35.616 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ cluster=cluster1 2023-08-26T13:57:35.616 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ instance=3 2023-08-26T13:57:35.616 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ '[' 3 = cluster1:3 ']' 2023-08-26T13:57:35.617 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ eval cluster=cluster1 2023-08-26T13:57:35.617 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ cluster=cluster1 2023-08-26T13:57:35.617 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ eval instance=3 2023-08-26T13:57:35.617 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ instance=3 2023-08-26T13:57:35.617 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ daemon_asok_file cluster1:3 cluster1 2023-08-26T13:57:35.617 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ local local_cluster=cluster1:3 2023-08-26T13:57:35.618 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ local cluster=cluster1 2023-08-26T13:57:35.618 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ local instance 2023-08-26T13:57:35.618 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ set_cluster_instance cluster1:3 local_cluster instance 2023-08-26T13:57:35.618 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ local val=cluster1:3 2023-08-26T13:57:35.618 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ local cluster_var_name=local_cluster 2023-08-26T13:57:35.619 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ local instance_var_name=instance 2023-08-26T13:57:35.619 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ cluster=cluster1 2023-08-26T13:57:35.619 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ instance=3 2023-08-26T13:57:35.619 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ '[' 3 = cluster1:3 ']' 2023-08-26T13:57:35.619 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ eval local_cluster=cluster1 2023-08-26T13:57:35.619 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++++ local_cluster=cluster1 2023-08-26T13:57:35.620 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ eval instance=3 2023-08-26T13:57:35.620 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++++ instance=3 2023-08-26T13:57:35.620 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2023-08-26T13:57:35.634 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2023-08-26T13:57:35.634 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2023-08-26T13:57:35.634 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2023-08-26T13:57:35.634 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror status mirror/image_37 2023-08-26T13:57:35.747 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ return 0 2023-08-26T13:57:35.748 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ status_result=' "state": "Replaying"' 2023-08-26T13:57:35.748 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ echo ' "state": "Replaying"' 2023-08-26T13:57:35.749 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ grep -i Replaying 2023-08-26T13:57:35.749 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: "state": "Replaying" 2023-08-26T13:57:35.751 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ current_state=started 2023-08-26T13:57:35.751 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ test started = started 2023-08-26T13:57:35.751 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ return 0 2023-08-26T13:57:35.752 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_37 2023-08-26T13:57:35.752 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local local_cluster=cluster1 2023-08-26T13:57:35.752 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster2 2023-08-26T13:57:35.752 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T13:57:35.752 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=image_37 2023-08-26T13:57:35.753 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ '[' snapshot = journal ']' 2023-08-26T13:57:35.753 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ '[' snapshot = snapshot ']' 2023-08-26T13:57:35.753 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_37 2023-08-26T13:57:35.753 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local local_cluster=cluster1 2023-08-26T13:57:35.753 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster2 2023-08-26T13:57:35.753 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T13:57:35.754 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=image_37 2023-08-26T13:57:35.754 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ mkfname cluster2-mirror-image_37.status 2023-08-26T13:57:35.754 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ echo cluster2-mirror-image_37.status 2023-08-26T13:57:35.754 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ sed -e 's|[/ ]|_|g' 2023-08-26T13:57:35.754 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local status_log=/tmp/tmp.TfDn6BcQ3N/cluster2-mirror-image_37.status 2023-08-26T13:57:35.754 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ mkfname cluster1-mirror-image_37.status 2023-08-26T13:57:35.755 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ echo cluster1-mirror-image_37.status 2023-08-26T13:57:35.755 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ sed -e 's|[/ ]|_|g' 2023-08-26T13:57:35.756 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local local_status_log=/tmp/tmp.TfDn6BcQ3N/cluster1-mirror-image_37.status 2023-08-26T13:57:35.756 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ mirror_image_snapshot cluster2 mirror image_37 2023-08-26T13:57:35.756 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster2 2023-08-26T13:57:35.756 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T13:57:35.756 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=image_37 2023-08-26T13:57:35.757 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_37 2023-08-26T13:57:36.227 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:Snapshot ID: 191 2023-08-26T13:57:36.235 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_37 /tmp/tmp.TfDn6BcQ3N/cluster2-mirror-image_37.status 2023-08-26T13:57:36.236 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster2 2023-08-26T13:57:36.236 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T13:57:36.236 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=image_37 2023-08-26T13:57:36.236 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local log=/tmp/tmp.TfDn6BcQ3N/cluster2-mirror-image_37.status 2023-08-26T13:57:36.237 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_37 --format xml 2023-08-26T13:57:36.237 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-08-26T13:57:36.278 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ xmlstarlet sel -t -v //snapshot/id 2023-08-26T13:57:36.280 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local snapshot_id=191 2023-08-26T13:57:36.280 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ true 2023-08-26T13:57:36.280 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-08-26T13:57:36.280 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ sleep 0.2 2023-08-26T13:57:36.481 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_37 /tmp/tmp.TfDn6BcQ3N/cluster1-mirror-image_37.status 2023-08-26T13:57:36.481 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster1 2023-08-26T13:57:36.482 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T13:57:36.482 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=image_37 2023-08-26T13:57:36.482 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local log=/tmp/tmp.TfDn6BcQ3N/cluster1-mirror-image_37.status 2023-08-26T13:57:36.482 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_37 --format xml 2023-08-26T13:57:36.482 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-08-26T13:57:36.527 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ true 2023-08-26T13:57:36.527 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-08-26T13:57:36.529 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:-:1.1: Document is empty 2023-08-26T13:57:36.529 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr: 2023-08-26T13:57:36.529 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:^ 2023-08-26T13:57:36.529 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local primary_snapshot_id= 2023-08-26T13:57:36.529 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ test 191 = '' 2023-08-26T13:57:36.530 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-08-26T13:57:36.530 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ sleep 0.4 2023-08-26T13:57:36.930 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_37 /tmp/tmp.TfDn6BcQ3N/cluster1-mirror-image_37.status 2023-08-26T13:57:36.931 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster1 2023-08-26T13:57:36.931 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T13:57:36.931 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=image_37 2023-08-26T13:57:36.931 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local log=/tmp/tmp.TfDn6BcQ3N/cluster1-mirror-image_37.status 2023-08-26T13:57:36.931 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_37 --format xml 2023-08-26T13:57:36.931 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-08-26T13:57:36.977 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ true 2023-08-26T13:57:36.977 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-08-26T13:57:36.978 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:-:1.1: Document is empty 2023-08-26T13:57:36.979 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr: 2023-08-26T13:57:36.979 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:^ 2023-08-26T13:57:36.979 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local primary_snapshot_id= 2023-08-26T13:57:36.980 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ test 191 = '' 2023-08-26T13:57:36.980 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-08-26T13:57:36.980 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ sleep 0.8 2023-08-26T13:57:37.780 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_37 /tmp/tmp.TfDn6BcQ3N/cluster1-mirror-image_37.status 2023-08-26T13:57:37.781 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster1 2023-08-26T13:57:37.781 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T13:57:37.781 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=image_37 2023-08-26T13:57:37.781 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local log=/tmp/tmp.TfDn6BcQ3N/cluster1-mirror-image_37.status 2023-08-26T13:57:37.781 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_37 --format xml 2023-08-26T13:57:37.782 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-08-26T13:57:37.826 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-08-26T13:57:37.828 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local primary_snapshot_id=191 2023-08-26T13:57:37.828 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ test 191 = 191 2023-08-26T13:57:37.829 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ return 0 2023-08-26T13:57:37.829 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ wait_for_snap_present cluster1 mirror image_37 snap 2023-08-26T13:57:37.829 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster1 2023-08-26T13:57:37.829 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T13:57:37.829 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=image_37 2023-08-26T13:57:37.830 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local snap_name=snap 2023-08-26T13:57:37.830 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local s 2023-08-26T13:57:37.830 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2023-08-26T13:57:37.830 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ sleep 1 2023-08-26T13:57:38.829 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rbd --cluster cluster1 info mirror/image_37@snap 2023-08-26T13:57:38.876 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:rbd image 'image_37': 2023-08-26T13:57:38.876 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: size 128 MiB in 32 objects 2023-08-26T13:57:38.876 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: order 22 (4 MiB objects) 2023-08-26T13:57:38.877 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: snapshot_count: 2 2023-08-26T13:57:38.877 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: id: 10b7ed814206 2023-08-26T13:57:38.877 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: block_name_prefix: rbd_data.10b7ed814206 2023-08-26T13:57:38.877 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: format: 2 2023-08-26T13:57:38.877 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: features: layering, exclusive-lock, non-primary 2023-08-26T13:57:38.878 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: op_features: 2023-08-26T13:57:38.878 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: flags: 2023-08-26T13:57:38.878 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: create_timestamp: Sat Aug 26 13:48:51 2023 2023-08-26T13:57:38.878 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: access_timestamp: Sat Aug 26 13:48:51 2023 2023-08-26T13:57:38.878 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: modify_timestamp: Sat Aug 26 13:48:51 2023 2023-08-26T13:57:38.879 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: protected: False 2023-08-26T13:57:38.879 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: mirroring state: enabled 2023-08-26T13:57:38.879 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: mirroring mode: snapshot 2023-08-26T13:57:38.879 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: mirroring global id: 339b230b-a92d-47ff-8812-3e7a71f8b968 2023-08-26T13:57:38.879 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: mirroring primary: false 2023-08-26T13:57:38.880 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ return 0 2023-08-26T13:57:38.881 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ compare_image_snaps mirror image_37 snap 2023-08-26T13:57:38.881 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T13:57:38.881 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=image_37 2023-08-26T13:57:38.881 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local snap_name=snap 2023-08-26T13:57:38.881 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local ret=0 2023-08-26T13:57:38.882 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local rmt_export=/tmp/tmp.TfDn6BcQ3N/cluster2-mirror-image_37.export 2023-08-26T13:57:38.882 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local loc_export=/tmp/tmp.TfDn6BcQ3N/cluster1-mirror-image_37.export 2023-08-26T13:57:38.882 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rm -f /tmp/tmp.TfDn6BcQ3N/cluster2-mirror-image_37.export /tmp/tmp.TfDn6BcQ3N/cluster1-mirror-image_37.export 2023-08-26T13:57:38.882 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rbd --cluster cluster2 -p mirror export image_37@snap /tmp/tmp.TfDn6BcQ3N/cluster2-mirror-image_37.export 2023-08-26T13:57:39.060 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-08-26T13:57:39.065 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rbd --cluster cluster1 -p mirror export image_37@snap /tmp/tmp.TfDn6BcQ3N/cluster1-mirror-image_37.export 2023-08-26T13:57:39.306 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-08-26T13:57:39.312 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ cmp /tmp/tmp.TfDn6BcQ3N/cluster2-mirror-image_37.export /tmp/tmp.TfDn6BcQ3N/cluster1-mirror-image_37.export 2023-08-26T13:57:39.367 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rm -f /tmp/tmp.TfDn6BcQ3N/cluster2-mirror-image_37.export /tmp/tmp.TfDn6BcQ3N/cluster1-mirror-image_37.export 2023-08-26T13:57:39.404 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ return 0 2023-08-26T13:57:39.404 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-08-26T13:57:39.404 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ image=image_38 2023-08-26T13:57:39.405 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ create_snap cluster2 mirror image_38 snap 2023-08-26T13:57:39.405 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster2 2023-08-26T13:57:39.405 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T13:57:39.405 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=image_38 2023-08-26T13:57:39.405 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local snap_name=snap 2023-08-26T13:57:39.406 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rbd --cluster cluster2 -p mirror snap create image_38@snap --debug-rbd=20 --debug-journaler=20 2023-08-26T13:57:41.233 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ wait_for_image_replay_started cluster1 mirror image_38 2023-08-26T13:57:41.233 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster1 2023-08-26T13:57:41.234 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T13:57:41.234 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=image_38 2023-08-26T13:57:41.234 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ wait_for_image_replay_state cluster1 mirror image_38 started 2023-08-26T13:57:41.234 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster1 2023-08-26T13:57:41.234 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T13:57:41.235 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=image_38 2023-08-26T13:57:41.235 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local state=started 2023-08-26T13:57:41.235 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local s 2023-08-26T13:57:41.235 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-08-26T13:57:41.235 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ sleep 1 2023-08-26T13:57:42.234 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ test_image_replay_state cluster1 mirror image_38 started 2023-08-26T13:57:42.235 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster1 2023-08-26T13:57:42.235 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T13:57:42.235 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=image_38 2023-08-26T13:57:42.235 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local test_state=started 2023-08-26T13:57:42.235 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local status_result 2023-08-26T13:57:42.236 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local current_state=stopped 2023-08-26T13:57:42.239 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_38 2023-08-26T13:57:42.239 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local cluster_instance=cluster1 2023-08-26T13:57:42.240 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ shift 2023-08-26T13:57:42.240 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local cluster=cluster1 2023-08-26T13:57:42.240 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ grep -i state 2023-08-26T13:57:42.240 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local instance=cluster1 2023-08-26T13:57:42.240 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local loop_instance 2023-08-26T13:57:42.240 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-08-26T13:57:42.241 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ sleep 0 2023-08-26T13:57:42.241 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ '[' cluster1 '!=' cluster1 ']' 2023-08-26T13:57:42.241 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ seq 0 3 2023-08-26T13:57:42.241 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-08-26T13:57:42.241 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_38 2023-08-26T13:57:42.242 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local cluster=cluster1:0 2023-08-26T13:57:42.242 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ shift 2023-08-26T13:57:42.242 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local instance 2023-08-26T13:57:42.242 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ set_cluster_instance cluster1:0 cluster instance 2023-08-26T13:57:42.242 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local val=cluster1:0 2023-08-26T13:57:42.242 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local cluster_var_name=cluster 2023-08-26T13:57:42.243 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local instance_var_name=instance 2023-08-26T13:57:42.243 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ cluster=cluster1 2023-08-26T13:57:42.243 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ instance=0 2023-08-26T13:57:42.243 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ '[' 0 = cluster1:0 ']' 2023-08-26T13:57:42.243 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ eval cluster=cluster1 2023-08-26T13:57:42.244 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ cluster=cluster1 2023-08-26T13:57:42.244 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ eval instance=0 2023-08-26T13:57:42.244 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ instance=0 2023-08-26T13:57:42.244 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ daemon_asok_file cluster1:0 cluster1 2023-08-26T13:57:42.244 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ local local_cluster=cluster1:0 2023-08-26T13:57:42.245 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ local cluster=cluster1 2023-08-26T13:57:42.245 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ local instance 2023-08-26T13:57:42.245 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2023-08-26T13:57:42.245 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ local val=cluster1:0 2023-08-26T13:57:42.245 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ local cluster_var_name=local_cluster 2023-08-26T13:57:42.245 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ local instance_var_name=instance 2023-08-26T13:57:42.246 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ cluster=cluster1 2023-08-26T13:57:42.246 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ instance=0 2023-08-26T13:57:42.246 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ '[' 0 = cluster1:0 ']' 2023-08-26T13:57:42.246 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ eval local_cluster=cluster1 2023-08-26T13:57:42.246 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++++ local_cluster=cluster1 2023-08-26T13:57:42.246 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ eval instance=0 2023-08-26T13:57:42.247 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++++ instance=0 2023-08-26T13:57:42.247 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2023-08-26T13:57:42.260 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-08-26T13:57:42.260 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-08-26T13:57:42.260 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-08-26T13:57:42.260 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_38 2023-08-26T13:57:42.369 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:Invalid command: unused arguments: ['mirror/image_38'] 2023-08-26T13:57:42.370 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:rbd mirror status : get status for rbd mirror 2023-08-26T13:57:42.370 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:admin_socket: invalid command 2023-08-26T13:57:42.372 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-08-26T13:57:42.373 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_38 2023-08-26T13:57:42.373 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local cluster=cluster1:1 2023-08-26T13:57:42.373 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ shift 2023-08-26T13:57:42.373 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local instance 2023-08-26T13:57:42.373 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ set_cluster_instance cluster1:1 cluster instance 2023-08-26T13:57:42.374 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local val=cluster1:1 2023-08-26T13:57:42.374 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local cluster_var_name=cluster 2023-08-26T13:57:42.374 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local instance_var_name=instance 2023-08-26T13:57:42.374 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ cluster=cluster1 2023-08-26T13:57:42.374 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ instance=1 2023-08-26T13:57:42.375 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ '[' 1 = cluster1:1 ']' 2023-08-26T13:57:42.375 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ eval cluster=cluster1 2023-08-26T13:57:42.375 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ cluster=cluster1 2023-08-26T13:57:42.375 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ eval instance=1 2023-08-26T13:57:42.375 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ instance=1 2023-08-26T13:57:42.375 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ daemon_asok_file cluster1:1 cluster1 2023-08-26T13:57:42.376 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ local local_cluster=cluster1:1 2023-08-26T13:57:42.376 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ local cluster=cluster1 2023-08-26T13:57:42.376 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ local instance 2023-08-26T13:57:42.376 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2023-08-26T13:57:42.376 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ local val=cluster1:1 2023-08-26T13:57:42.377 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ local cluster_var_name=local_cluster 2023-08-26T13:57:42.377 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ local instance_var_name=instance 2023-08-26T13:57:42.377 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ cluster=cluster1 2023-08-26T13:57:42.377 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ instance=1 2023-08-26T13:57:42.377 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ '[' 1 = cluster1:1 ']' 2023-08-26T13:57:42.377 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ eval local_cluster=cluster1 2023-08-26T13:57:42.378 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++++ local_cluster=cluster1 2023-08-26T13:57:42.378 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ eval instance=1 2023-08-26T13:57:42.378 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++++ instance=1 2023-08-26T13:57:42.378 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2023-08-26T13:57:42.391 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-08-26T13:57:42.391 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-08-26T13:57:42.391 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-08-26T13:57:42.391 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_38 2023-08-26T13:57:42.498 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:Invalid command: unused arguments: ['mirror/image_38'] 2023-08-26T13:57:42.498 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:rbd mirror status : get status for rbd mirror 2023-08-26T13:57:42.498 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:admin_socket: invalid command 2023-08-26T13:57:42.501 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-08-26T13:57:42.501 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/image_38 2023-08-26T13:57:42.502 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local cluster=cluster1:2 2023-08-26T13:57:42.502 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ shift 2023-08-26T13:57:42.502 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local instance 2023-08-26T13:57:42.502 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ set_cluster_instance cluster1:2 cluster instance 2023-08-26T13:57:42.502 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local val=cluster1:2 2023-08-26T13:57:42.502 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local cluster_var_name=cluster 2023-08-26T13:57:42.503 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local instance_var_name=instance 2023-08-26T13:57:42.503 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ cluster=cluster1 2023-08-26T13:57:42.503 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ instance=2 2023-08-26T13:57:42.503 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ '[' 2 = cluster1:2 ']' 2023-08-26T13:57:42.503 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ eval cluster=cluster1 2023-08-26T13:57:42.504 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ cluster=cluster1 2023-08-26T13:57:42.504 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ eval instance=2 2023-08-26T13:57:42.504 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ instance=2 2023-08-26T13:57:42.504 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ daemon_asok_file cluster1:2 cluster1 2023-08-26T13:57:42.504 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ local local_cluster=cluster1:2 2023-08-26T13:57:42.505 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ local cluster=cluster1 2023-08-26T13:57:42.505 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ local instance 2023-08-26T13:57:42.505 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2023-08-26T13:57:42.505 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ local val=cluster1:2 2023-08-26T13:57:42.505 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ local cluster_var_name=local_cluster 2023-08-26T13:57:42.505 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ local instance_var_name=instance 2023-08-26T13:57:42.506 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ cluster=cluster1 2023-08-26T13:57:42.506 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ instance=2 2023-08-26T13:57:42.506 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ '[' 2 = cluster1:2 ']' 2023-08-26T13:57:42.506 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ eval local_cluster=cluster1 2023-08-26T13:57:42.506 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++++ local_cluster=cluster1 2023-08-26T13:57:42.507 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ eval instance=2 2023-08-26T13:57:42.507 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++++ instance=2 2023-08-26T13:57:42.507 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2023-08-26T13:57:42.519 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-08-26T13:57:42.519 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-08-26T13:57:42.520 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-08-26T13:57:42.520 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/image_38 2023-08-26T13:57:42.632 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ return 0 2023-08-26T13:57:42.633 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ status_result=' "state": "Replaying"' 2023-08-26T13:57:42.633 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ echo ' "state": "Replaying"' 2023-08-26T13:57:42.633 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ grep -i Replaying 2023-08-26T13:57:42.633 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: "state": "Replaying" 2023-08-26T13:57:42.636 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ current_state=started 2023-08-26T13:57:42.636 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ test started = started 2023-08-26T13:57:42.636 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ return 0 2023-08-26T13:57:42.636 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_38 2023-08-26T13:57:42.637 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local local_cluster=cluster1 2023-08-26T13:57:42.637 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster2 2023-08-26T13:57:42.637 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T13:57:42.637 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=image_38 2023-08-26T13:57:42.637 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ '[' snapshot = journal ']' 2023-08-26T13:57:42.638 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ '[' snapshot = snapshot ']' 2023-08-26T13:57:42.638 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_38 2023-08-26T13:57:42.638 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local local_cluster=cluster1 2023-08-26T13:57:42.638 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster2 2023-08-26T13:57:42.638 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T13:57:42.639 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=image_38 2023-08-26T13:57:42.639 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ mkfname cluster2-mirror-image_38.status 2023-08-26T13:57:42.639 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ echo cluster2-mirror-image_38.status 2023-08-26T13:57:42.639 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ sed -e 's|[/ ]|_|g' 2023-08-26T13:57:42.639 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local status_log=/tmp/tmp.TfDn6BcQ3N/cluster2-mirror-image_38.status 2023-08-26T13:57:42.640 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ mkfname cluster1-mirror-image_38.status 2023-08-26T13:57:42.640 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ echo cluster1-mirror-image_38.status 2023-08-26T13:57:42.640 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ sed -e 's|[/ ]|_|g' 2023-08-26T13:57:42.641 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local local_status_log=/tmp/tmp.TfDn6BcQ3N/cluster1-mirror-image_38.status 2023-08-26T13:57:42.641 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ mirror_image_snapshot cluster2 mirror image_38 2023-08-26T13:57:42.641 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster2 2023-08-26T13:57:42.641 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T13:57:42.642 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=image_38 2023-08-26T13:57:42.642 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_38 2023-08-26T13:57:43.244 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:Snapshot ID: 193 2023-08-26T13:57:43.253 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_38 /tmp/tmp.TfDn6BcQ3N/cluster2-mirror-image_38.status 2023-08-26T13:57:43.253 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster2 2023-08-26T13:57:43.253 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T13:57:43.253 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=image_38 2023-08-26T13:57:43.253 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local log=/tmp/tmp.TfDn6BcQ3N/cluster2-mirror-image_38.status 2023-08-26T13:57:43.254 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_38 --format xml 2023-08-26T13:57:43.254 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-08-26T13:57:43.296 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ xmlstarlet sel -t -v //snapshot/id 2023-08-26T13:57:43.298 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local snapshot_id=193 2023-08-26T13:57:43.298 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ true 2023-08-26T13:57:43.298 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-08-26T13:57:43.298 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ sleep 0.2 2023-08-26T13:57:43.499 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_38 /tmp/tmp.TfDn6BcQ3N/cluster1-mirror-image_38.status 2023-08-26T13:57:43.499 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster1 2023-08-26T13:57:43.500 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T13:57:43.500 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=image_38 2023-08-26T13:57:43.500 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local log=/tmp/tmp.TfDn6BcQ3N/cluster1-mirror-image_38.status 2023-08-26T13:57:43.500 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_38 --format xml 2023-08-26T13:57:43.500 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-08-26T13:57:43.551 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ true 2023-08-26T13:57:43.551 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-08-26T13:57:43.552 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:-:1.1: Document is empty 2023-08-26T13:57:43.552 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr: 2023-08-26T13:57:43.552 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:^ 2023-08-26T13:57:43.553 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local primary_snapshot_id= 2023-08-26T13:57:43.553 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ test 193 = '' 2023-08-26T13:57:43.553 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-08-26T13:57:43.554 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ sleep 0.4 2023-08-26T13:57:43.954 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_38 /tmp/tmp.TfDn6BcQ3N/cluster1-mirror-image_38.status 2023-08-26T13:57:43.954 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster1 2023-08-26T13:57:43.954 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T13:57:43.955 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=image_38 2023-08-26T13:57:43.955 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local log=/tmp/tmp.TfDn6BcQ3N/cluster1-mirror-image_38.status 2023-08-26T13:57:43.955 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_38 --format xml 2023-08-26T13:57:43.955 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-08-26T13:57:44.001 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ true 2023-08-26T13:57:44.003 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-08-26T13:57:44.003 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:-:1.1: Document is empty 2023-08-26T13:57:44.004 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr: 2023-08-26T13:57:44.004 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:^ 2023-08-26T13:57:44.004 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local primary_snapshot_id= 2023-08-26T13:57:44.004 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ test 193 = '' 2023-08-26T13:57:44.004 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-08-26T13:57:44.005 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ sleep 0.8 2023-08-26T13:57:44.805 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_38 /tmp/tmp.TfDn6BcQ3N/cluster1-mirror-image_38.status 2023-08-26T13:57:44.805 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster1 2023-08-26T13:57:44.806 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T13:57:44.806 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=image_38 2023-08-26T13:57:44.806 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local log=/tmp/tmp.TfDn6BcQ3N/cluster1-mirror-image_38.status 2023-08-26T13:57:44.806 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_38 --format xml 2023-08-26T13:57:44.806 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-08-26T13:57:44.857 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-08-26T13:57:44.859 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local primary_snapshot_id=193 2023-08-26T13:57:44.860 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ test 193 = 193 2023-08-26T13:57:44.860 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ return 0 2023-08-26T13:57:44.860 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ wait_for_snap_present cluster1 mirror image_38 snap 2023-08-26T13:57:44.860 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster1 2023-08-26T13:57:44.861 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T13:57:44.861 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=image_38 2023-08-26T13:57:44.861 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local snap_name=snap 2023-08-26T13:57:44.861 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local s 2023-08-26T13:57:44.861 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2023-08-26T13:57:44.862 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ sleep 1 2023-08-26T13:57:45.860 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rbd --cluster cluster1 info mirror/image_38@snap 2023-08-26T13:57:45.907 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:rbd image 'image_38': 2023-08-26T13:57:45.907 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: size 128 MiB in 32 objects 2023-08-26T13:57:45.908 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: order 22 (4 MiB objects) 2023-08-26T13:57:45.908 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: snapshot_count: 2 2023-08-26T13:57:45.908 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: id: 122b6ae67b0e 2023-08-26T13:57:45.908 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: block_name_prefix: rbd_data.122b6ae67b0e 2023-08-26T13:57:45.908 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: format: 2 2023-08-26T13:57:45.909 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: features: layering, exclusive-lock, non-primary 2023-08-26T13:57:45.909 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: op_features: 2023-08-26T13:57:45.909 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: flags: 2023-08-26T13:57:45.909 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: create_timestamp: Sat Aug 26 13:48:58 2023 2023-08-26T13:57:45.909 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: access_timestamp: Sat Aug 26 13:48:58 2023 2023-08-26T13:57:45.909 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: modify_timestamp: Sat Aug 26 13:48:58 2023 2023-08-26T13:57:45.910 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: protected: False 2023-08-26T13:57:45.910 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: mirroring state: enabled 2023-08-26T13:57:45.910 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: mirroring mode: snapshot 2023-08-26T13:57:45.910 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: mirroring global id: 1e20d016-f50e-43c9-b2e8-47e995fee9bd 2023-08-26T13:57:45.910 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: mirroring primary: false 2023-08-26T13:57:45.912 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ return 0 2023-08-26T13:57:45.912 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ compare_image_snaps mirror image_38 snap 2023-08-26T13:57:45.912 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T13:57:45.912 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=image_38 2023-08-26T13:57:45.912 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local snap_name=snap 2023-08-26T13:57:45.913 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local ret=0 2023-08-26T13:57:45.913 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local rmt_export=/tmp/tmp.TfDn6BcQ3N/cluster2-mirror-image_38.export 2023-08-26T13:57:45.913 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local loc_export=/tmp/tmp.TfDn6BcQ3N/cluster1-mirror-image_38.export 2023-08-26T13:57:45.913 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rm -f /tmp/tmp.TfDn6BcQ3N/cluster2-mirror-image_38.export /tmp/tmp.TfDn6BcQ3N/cluster1-mirror-image_38.export 2023-08-26T13:57:45.913 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rbd --cluster cluster2 -p mirror export image_38@snap /tmp/tmp.TfDn6BcQ3N/cluster2-mirror-image_38.export 2023-08-26T13:57:46.098 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-08-26T13:57:46.103 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rbd --cluster cluster1 -p mirror export image_38@snap /tmp/tmp.TfDn6BcQ3N/cluster1-mirror-image_38.export 2023-08-26T13:57:46.333 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-08-26T13:57:46.339 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ cmp /tmp/tmp.TfDn6BcQ3N/cluster2-mirror-image_38.export /tmp/tmp.TfDn6BcQ3N/cluster1-mirror-image_38.export 2023-08-26T13:57:46.394 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rm -f /tmp/tmp.TfDn6BcQ3N/cluster2-mirror-image_38.export /tmp/tmp.TfDn6BcQ3N/cluster1-mirror-image_38.export 2023-08-26T13:57:46.431 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ return 0 2023-08-26T13:57:46.431 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-08-26T13:57:46.431 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ image=image_39 2023-08-26T13:57:46.432 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ create_snap cluster2 mirror image_39 snap 2023-08-26T13:57:46.432 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster2 2023-08-26T13:57:46.432 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T13:57:46.432 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=image_39 2023-08-26T13:57:46.432 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local snap_name=snap 2023-08-26T13:57:46.433 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rbd --cluster cluster2 -p mirror snap create image_39@snap --debug-rbd=20 --debug-journaler=20 2023-08-26T13:57:46.759 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.0 with skip (0.9979523523500067) > weight (0.25) 2023-08-26T13:57:46.759 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.1 with skip (0.7685920172114983) > weight (0.25) 2023-08-26T13:57:46.759 INFO:tasks.rbd_mirror_thrash:kill cluster1.client.mirror.2 2023-08-26T13:57:46.760 INFO:tasks.rbd_mirror.cluster1.client.mirror.2:Sent signal 15 2023-08-26T13:57:46.760 INFO:tasks.rbd_mirror_thrash:waiting for 3 secs before reviving daemons 2023-08-26T13:57:46.761 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi124.stderr:2023-08-26T13:57:46.759+0000 7fa57eae4700 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.2 (PID: 125925) UID: 1000 2023-08-26T13:57:48.250 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ wait_for_image_replay_started cluster1 mirror image_39 2023-08-26T13:57:48.251 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster1 2023-08-26T13:57:48.251 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T13:57:48.251 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=image_39 2023-08-26T13:57:48.251 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ wait_for_image_replay_state cluster1 mirror image_39 started 2023-08-26T13:57:48.251 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster1 2023-08-26T13:57:48.251 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T13:57:48.252 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=image_39 2023-08-26T13:57:48.252 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local state=started 2023-08-26T13:57:48.252 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local s 2023-08-26T13:57:48.252 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-08-26T13:57:48.252 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ sleep 1 2023-08-26T13:57:49.251 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ test_image_replay_state cluster1 mirror image_39 started 2023-08-26T13:57:49.252 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster1 2023-08-26T13:57:49.252 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T13:57:49.252 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=image_39 2023-08-26T13:57:49.252 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local test_state=started 2023-08-26T13:57:49.253 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local status_result 2023-08-26T13:57:49.253 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local current_state=stopped 2023-08-26T13:57:49.254 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_39 2023-08-26T13:57:49.254 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local cluster_instance=cluster1 2023-08-26T13:57:49.254 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ shift 2023-08-26T13:57:49.254 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local cluster=cluster1 2023-08-26T13:57:49.255 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local instance=cluster1 2023-08-26T13:57:49.255 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ grep -i state 2023-08-26T13:57:49.255 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local loop_instance 2023-08-26T13:57:49.255 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-08-26T13:57:49.255 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ sleep 0 2023-08-26T13:57:49.256 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ '[' cluster1 '!=' cluster1 ']' 2023-08-26T13:57:49.256 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ seq 0 3 2023-08-26T13:57:49.258 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-08-26T13:57:49.258 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_39 2023-08-26T13:57:49.259 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local cluster=cluster1:0 2023-08-26T13:57:49.259 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ shift 2023-08-26T13:57:49.259 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local instance 2023-08-26T13:57:49.259 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ set_cluster_instance cluster1:0 cluster instance 2023-08-26T13:57:49.259 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local val=cluster1:0 2023-08-26T13:57:49.259 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local cluster_var_name=cluster 2023-08-26T13:57:49.260 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local instance_var_name=instance 2023-08-26T13:57:49.260 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ cluster=cluster1 2023-08-26T13:57:49.260 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ instance=0 2023-08-26T13:57:49.260 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ '[' 0 = cluster1:0 ']' 2023-08-26T13:57:49.260 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ eval cluster=cluster1 2023-08-26T13:57:49.261 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ cluster=cluster1 2023-08-26T13:57:49.261 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ eval instance=0 2023-08-26T13:57:49.261 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ instance=0 2023-08-26T13:57:49.261 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ daemon_asok_file cluster1:0 cluster1 2023-08-26T13:57:49.261 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ local local_cluster=cluster1:0 2023-08-26T13:57:49.261 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ local cluster=cluster1 2023-08-26T13:57:49.262 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ local instance 2023-08-26T13:57:49.262 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2023-08-26T13:57:49.262 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ local val=cluster1:0 2023-08-26T13:57:49.262 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ local cluster_var_name=local_cluster 2023-08-26T13:57:49.262 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ local instance_var_name=instance 2023-08-26T13:57:49.262 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ cluster=cluster1 2023-08-26T13:57:49.263 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ instance=0 2023-08-26T13:57:49.263 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ '[' 0 = cluster1:0 ']' 2023-08-26T13:57:49.263 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ eval local_cluster=cluster1 2023-08-26T13:57:49.263 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++++ local_cluster=cluster1 2023-08-26T13:57:49.263 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ eval instance=0 2023-08-26T13:57:49.263 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++++ instance=0 2023-08-26T13:57:49.264 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2023-08-26T13:57:49.278 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-08-26T13:57:49.278 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-08-26T13:57:49.278 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-08-26T13:57:49.278 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_39 2023-08-26T13:57:49.388 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:Invalid command: unused arguments: ['mirror/image_39'] 2023-08-26T13:57:49.389 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:rbd mirror status : get status for rbd mirror 2023-08-26T13:57:49.389 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:admin_socket: invalid command 2023-08-26T13:57:49.392 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-08-26T13:57:49.392 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_39 2023-08-26T13:57:49.393 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local cluster=cluster1:1 2023-08-26T13:57:49.393 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ shift 2023-08-26T13:57:49.393 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local instance 2023-08-26T13:57:49.393 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ set_cluster_instance cluster1:1 cluster instance 2023-08-26T13:57:49.393 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local val=cluster1:1 2023-08-26T13:57:49.393 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local cluster_var_name=cluster 2023-08-26T13:57:49.394 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local instance_var_name=instance 2023-08-26T13:57:49.394 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ cluster=cluster1 2023-08-26T13:57:49.394 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ instance=1 2023-08-26T13:57:49.394 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ '[' 1 = cluster1:1 ']' 2023-08-26T13:57:49.394 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ eval cluster=cluster1 2023-08-26T13:57:49.395 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ cluster=cluster1 2023-08-26T13:57:49.395 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ eval instance=1 2023-08-26T13:57:49.395 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ instance=1 2023-08-26T13:57:49.395 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ daemon_asok_file cluster1:1 cluster1 2023-08-26T13:57:49.395 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ local local_cluster=cluster1:1 2023-08-26T13:57:49.395 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ local cluster=cluster1 2023-08-26T13:57:49.396 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ local instance 2023-08-26T13:57:49.396 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2023-08-26T13:57:49.396 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ local val=cluster1:1 2023-08-26T13:57:49.396 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ local cluster_var_name=local_cluster 2023-08-26T13:57:49.396 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ local instance_var_name=instance 2023-08-26T13:57:49.397 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ cluster=cluster1 2023-08-26T13:57:49.397 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ instance=1 2023-08-26T13:57:49.397 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ '[' 1 = cluster1:1 ']' 2023-08-26T13:57:49.397 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ eval local_cluster=cluster1 2023-08-26T13:57:49.397 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++++ local_cluster=cluster1 2023-08-26T13:57:49.398 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ eval instance=1 2023-08-26T13:57:49.398 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++++ instance=1 2023-08-26T13:57:49.398 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2023-08-26T13:57:49.410 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-08-26T13:57:49.410 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-08-26T13:57:49.411 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-08-26T13:57:49.411 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_39 2023-08-26T13:57:49.524 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ return 0 2023-08-26T13:57:49.525 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ status_result=' "state": "Replaying"' 2023-08-26T13:57:49.526 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ echo ' "state": "Replaying"' 2023-08-26T13:57:49.526 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ grep -i Replaying 2023-08-26T13:57:49.526 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: "state": "Replaying" 2023-08-26T13:57:49.528 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ current_state=started 2023-08-26T13:57:49.528 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ test started = started 2023-08-26T13:57:49.528 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ return 0 2023-08-26T13:57:49.529 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_39 2023-08-26T13:57:49.529 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local local_cluster=cluster1 2023-08-26T13:57:49.529 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster2 2023-08-26T13:57:49.529 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T13:57:49.529 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=image_39 2023-08-26T13:57:49.530 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ '[' snapshot = journal ']' 2023-08-26T13:57:49.530 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ '[' snapshot = snapshot ']' 2023-08-26T13:57:49.530 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_39 2023-08-26T13:57:49.530 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local local_cluster=cluster1 2023-08-26T13:57:49.530 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster2 2023-08-26T13:57:49.531 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T13:57:49.531 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=image_39 2023-08-26T13:57:49.531 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ mkfname cluster2-mirror-image_39.status 2023-08-26T13:57:49.531 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ echo cluster2-mirror-image_39.status 2023-08-26T13:57:49.531 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ sed -e 's|[/ ]|_|g' 2023-08-26T13:57:49.532 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local status_log=/tmp/tmp.TfDn6BcQ3N/cluster2-mirror-image_39.status 2023-08-26T13:57:49.533 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ mkfname cluster1-mirror-image_39.status 2023-08-26T13:57:49.533 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ echo cluster1-mirror-image_39.status 2023-08-26T13:57:49.533 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ sed -e 's|[/ ]|_|g' 2023-08-26T13:57:49.533 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local local_status_log=/tmp/tmp.TfDn6BcQ3N/cluster1-mirror-image_39.status 2023-08-26T13:57:49.534 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ mirror_image_snapshot cluster2 mirror image_39 2023-08-26T13:57:49.534 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster2 2023-08-26T13:57:49.534 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T13:57:49.534 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=image_39 2023-08-26T13:57:49.534 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_39 2023-08-26T13:57:49.761 INFO:tasks.rbd_mirror_thrash:waiting for cluster1.client.mirror.2 2023-08-26T13:57:49.761 INFO:teuthology.orchestra.run:waiting for 600 2023-08-26T13:57:49.761 INFO:tasks.rbd_mirror_thrash:reviving cluster1.client.mirror.2 2023-08-26T13:57:49.762 INFO:tasks.rbd_mirror.cluster1.client.mirror.2:Restarting daemon 2023-08-26T13:57:49.762 DEBUG:teuthology.orchestra.run.smithi124:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.2 2023-08-26T13:57:49.765 INFO:tasks.rbd_mirror.cluster1.client.mirror.2:Started 2023-08-26T13:57:49.765 INFO:tasks.rbd_mirror_thrash:waiting for 99 secs before thrashing 2023-08-26T13:57:50.254 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:Snapshot ID: 195 2023-08-26T13:57:50.262 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_39 /tmp/tmp.TfDn6BcQ3N/cluster2-mirror-image_39.status 2023-08-26T13:57:50.262 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster2 2023-08-26T13:57:50.263 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T13:57:50.263 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=image_39 2023-08-26T13:57:50.263 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local log=/tmp/tmp.TfDn6BcQ3N/cluster2-mirror-image_39.status 2023-08-26T13:57:50.263 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_39 --format xml 2023-08-26T13:57:50.263 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-08-26T13:57:50.304 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ xmlstarlet sel -t -v //snapshot/id 2023-08-26T13:57:50.306 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local snapshot_id=195 2023-08-26T13:57:50.306 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ true 2023-08-26T13:57:50.306 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-08-26T13:57:50.307 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ sleep 0.2 2023-08-26T13:57:50.507 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_39 /tmp/tmp.TfDn6BcQ3N/cluster1-mirror-image_39.status 2023-08-26T13:57:50.508 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster1 2023-08-26T13:57:50.508 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T13:57:50.508 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=image_39 2023-08-26T13:57:50.508 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local log=/tmp/tmp.TfDn6BcQ3N/cluster1-mirror-image_39.status 2023-08-26T13:57:50.509 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_39 --format xml 2023-08-26T13:57:50.509 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-08-26T13:57:50.555 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ true 2023-08-26T13:57:50.555 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-08-26T13:57:50.556 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:-:1.1: Document is empty 2023-08-26T13:57:50.556 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr: 2023-08-26T13:57:50.556 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:^ 2023-08-26T13:57:50.557 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local primary_snapshot_id= 2023-08-26T13:57:50.557 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ test 195 = '' 2023-08-26T13:57:50.557 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-08-26T13:57:50.558 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ sleep 0.4 2023-08-26T13:57:50.958 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_39 /tmp/tmp.TfDn6BcQ3N/cluster1-mirror-image_39.status 2023-08-26T13:57:50.958 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster1 2023-08-26T13:57:50.959 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T13:57:50.959 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=image_39 2023-08-26T13:57:50.959 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local log=/tmp/tmp.TfDn6BcQ3N/cluster1-mirror-image_39.status 2023-08-26T13:57:50.959 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_39 --format xml 2023-08-26T13:57:50.959 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-08-26T13:57:51.004 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ true 2023-08-26T13:57:51.004 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-08-26T13:57:51.005 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:-:1.1: Document is empty 2023-08-26T13:57:51.006 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr: 2023-08-26T13:57:51.006 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:^ 2023-08-26T13:57:51.006 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local primary_snapshot_id= 2023-08-26T13:57:51.007 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ test 195 = '' 2023-08-26T13:57:51.007 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-08-26T13:57:51.007 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ sleep 0.8 2023-08-26T13:57:51.807 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_39 /tmp/tmp.TfDn6BcQ3N/cluster1-mirror-image_39.status 2023-08-26T13:57:51.808 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster1 2023-08-26T13:57:51.808 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T13:57:51.808 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=image_39 2023-08-26T13:57:51.808 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local log=/tmp/tmp.TfDn6BcQ3N/cluster1-mirror-image_39.status 2023-08-26T13:57:51.809 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_39 --format xml 2023-08-26T13:57:51.809 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-08-26T13:57:51.853 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-08-26T13:57:51.855 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local primary_snapshot_id=195 2023-08-26T13:57:51.855 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ test 195 = 195 2023-08-26T13:57:51.856 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ return 0 2023-08-26T13:57:51.856 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ wait_for_snap_present cluster1 mirror image_39 snap 2023-08-26T13:57:51.856 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster1 2023-08-26T13:57:51.856 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T13:57:51.856 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=image_39 2023-08-26T13:57:51.857 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local snap_name=snap 2023-08-26T13:57:51.857 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local s 2023-08-26T13:57:51.857 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2023-08-26T13:57:51.857 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ sleep 1 2023-08-26T13:57:52.856 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rbd --cluster cluster1 info mirror/image_39@snap 2023-08-26T13:57:52.902 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:rbd image 'image_39': 2023-08-26T13:57:52.902 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: size 128 MiB in 32 objects 2023-08-26T13:57:52.902 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: order 22 (4 MiB objects) 2023-08-26T13:57:52.903 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: snapshot_count: 2 2023-08-26T13:57:52.903 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: id: 12c03d9ce77 2023-08-26T13:57:52.903 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: block_name_prefix: rbd_data.12c03d9ce77 2023-08-26T13:57:52.903 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: format: 2 2023-08-26T13:57:52.904 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: features: layering, exclusive-lock, non-primary 2023-08-26T13:57:52.904 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: op_features: 2023-08-26T13:57:52.904 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: flags: 2023-08-26T13:57:52.904 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: create_timestamp: Sat Aug 26 13:49:08 2023 2023-08-26T13:57:52.904 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: access_timestamp: Sat Aug 26 13:49:08 2023 2023-08-26T13:57:52.905 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: modify_timestamp: Sat Aug 26 13:49:08 2023 2023-08-26T13:57:52.905 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: protected: False 2023-08-26T13:57:52.905 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: mirroring state: enabled 2023-08-26T13:57:52.905 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: mirroring mode: snapshot 2023-08-26T13:57:52.905 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: mirroring global id: e777bb16-afba-4c52-994a-6980c13f1760 2023-08-26T13:57:52.906 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: mirroring primary: false 2023-08-26T13:57:52.907 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ return 0 2023-08-26T13:57:52.907 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ compare_image_snaps mirror image_39 snap 2023-08-26T13:57:52.907 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T13:57:52.907 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=image_39 2023-08-26T13:57:52.907 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local snap_name=snap 2023-08-26T13:57:52.908 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local ret=0 2023-08-26T13:57:52.908 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local rmt_export=/tmp/tmp.TfDn6BcQ3N/cluster2-mirror-image_39.export 2023-08-26T13:57:52.908 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local loc_export=/tmp/tmp.TfDn6BcQ3N/cluster1-mirror-image_39.export 2023-08-26T13:57:52.908 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rm -f /tmp/tmp.TfDn6BcQ3N/cluster2-mirror-image_39.export /tmp/tmp.TfDn6BcQ3N/cluster1-mirror-image_39.export 2023-08-26T13:57:52.909 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rbd --cluster cluster2 -p mirror export image_39@snap /tmp/tmp.TfDn6BcQ3N/cluster2-mirror-image_39.export 2023-08-26T13:57:53.092 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-08-26T13:57:53.098 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rbd --cluster cluster1 -p mirror export image_39@snap /tmp/tmp.TfDn6BcQ3N/cluster1-mirror-image_39.export 2023-08-26T13:57:53.345 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-08-26T13:57:53.352 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ cmp /tmp/tmp.TfDn6BcQ3N/cluster2-mirror-image_39.export /tmp/tmp.TfDn6BcQ3N/cluster1-mirror-image_39.export 2023-08-26T13:57:53.407 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rm -f /tmp/tmp.TfDn6BcQ3N/cluster2-mirror-image_39.export /tmp/tmp.TfDn6BcQ3N/cluster1-mirror-image_39.export 2023-08-26T13:57:53.445 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ return 0 2023-08-26T13:57:53.445 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-08-26T13:57:53.445 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ image=image_40 2023-08-26T13:57:53.445 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ create_snap cluster2 mirror image_40 snap 2023-08-26T13:57:53.446 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster2 2023-08-26T13:57:53.446 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T13:57:53.446 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=image_40 2023-08-26T13:57:53.446 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local snap_name=snap 2023-08-26T13:57:53.446 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rbd --cluster cluster2 -p mirror snap create image_40@snap --debug-rbd=20 --debug-journaler=20 2023-08-26T13:57:55.262 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ wait_for_image_replay_started cluster1 mirror image_40 2023-08-26T13:57:55.263 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster1 2023-08-26T13:57:55.263 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T13:57:55.263 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=image_40 2023-08-26T13:57:55.264 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ wait_for_image_replay_state cluster1 mirror image_40 started 2023-08-26T13:57:55.264 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster1 2023-08-26T13:57:55.264 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T13:57:55.265 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=image_40 2023-08-26T13:57:55.265 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local state=started 2023-08-26T13:57:55.265 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local s 2023-08-26T13:57:55.265 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-08-26T13:57:55.265 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ sleep 1 2023-08-26T13:57:56.264 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ test_image_replay_state cluster1 mirror image_40 started 2023-08-26T13:57:56.264 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster1 2023-08-26T13:57:56.264 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T13:57:56.264 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=image_40 2023-08-26T13:57:56.265 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local test_state=started 2023-08-26T13:57:56.265 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local status_result 2023-08-26T13:57:56.265 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local current_state=stopped 2023-08-26T13:57:56.265 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_40 2023-08-26T13:57:56.265 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local cluster_instance=cluster1 2023-08-26T13:57:56.266 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ grep -i state 2023-08-26T13:57:56.266 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ shift 2023-08-26T13:57:56.266 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local cluster=cluster1 2023-08-26T13:57:56.266 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local instance=cluster1 2023-08-26T13:57:56.266 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local loop_instance 2023-08-26T13:57:56.267 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-08-26T13:57:56.267 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ sleep 0 2023-08-26T13:57:56.270 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ '[' cluster1 '!=' cluster1 ']' 2023-08-26T13:57:56.270 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ seq 0 3 2023-08-26T13:57:56.272 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-08-26T13:57:56.272 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_40 2023-08-26T13:57:56.272 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local cluster=cluster1:0 2023-08-26T13:57:56.272 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ shift 2023-08-26T13:57:56.273 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local instance 2023-08-26T13:57:56.273 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ set_cluster_instance cluster1:0 cluster instance 2023-08-26T13:57:56.273 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local val=cluster1:0 2023-08-26T13:57:56.273 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local cluster_var_name=cluster 2023-08-26T13:57:56.273 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local instance_var_name=instance 2023-08-26T13:57:56.274 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ cluster=cluster1 2023-08-26T13:57:56.274 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ instance=0 2023-08-26T13:57:56.274 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ '[' 0 = cluster1:0 ']' 2023-08-26T13:57:56.274 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ eval cluster=cluster1 2023-08-26T13:57:56.274 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ cluster=cluster1 2023-08-26T13:57:56.274 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ eval instance=0 2023-08-26T13:57:56.275 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ instance=0 2023-08-26T13:57:56.275 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ daemon_asok_file cluster1:0 cluster1 2023-08-26T13:57:56.275 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ local local_cluster=cluster1:0 2023-08-26T13:57:56.275 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ local cluster=cluster1 2023-08-26T13:57:56.275 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ local instance 2023-08-26T13:57:56.276 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2023-08-26T13:57:56.276 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ local val=cluster1:0 2023-08-26T13:57:56.276 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ local cluster_var_name=local_cluster 2023-08-26T13:57:56.276 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ local instance_var_name=instance 2023-08-26T13:57:56.276 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ cluster=cluster1 2023-08-26T13:57:56.276 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ instance=0 2023-08-26T13:57:56.277 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ '[' 0 = cluster1:0 ']' 2023-08-26T13:57:56.277 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ eval local_cluster=cluster1 2023-08-26T13:57:56.277 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++++ local_cluster=cluster1 2023-08-26T13:57:56.277 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ eval instance=0 2023-08-26T13:57:56.277 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++++ instance=0 2023-08-26T13:57:56.278 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2023-08-26T13:57:56.289 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-08-26T13:57:56.290 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-08-26T13:57:56.290 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-08-26T13:57:56.290 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_40 2023-08-26T13:57:56.400 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:Invalid command: unused arguments: ['mirror/image_40'] 2023-08-26T13:57:56.400 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:rbd mirror status : get status for rbd mirror 2023-08-26T13:57:56.400 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:admin_socket: invalid command 2023-08-26T13:57:56.403 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-08-26T13:57:56.403 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_40 2023-08-26T13:57:56.404 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local cluster=cluster1:1 2023-08-26T13:57:56.404 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ shift 2023-08-26T13:57:56.404 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local instance 2023-08-26T13:57:56.404 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ set_cluster_instance cluster1:1 cluster instance 2023-08-26T13:57:56.404 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local val=cluster1:1 2023-08-26T13:57:56.404 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local cluster_var_name=cluster 2023-08-26T13:57:56.405 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local instance_var_name=instance 2023-08-26T13:57:56.405 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ cluster=cluster1 2023-08-26T13:57:56.405 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ instance=1 2023-08-26T13:57:56.405 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ '[' 1 = cluster1:1 ']' 2023-08-26T13:57:56.405 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ eval cluster=cluster1 2023-08-26T13:57:56.405 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ cluster=cluster1 2023-08-26T13:57:56.406 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ eval instance=1 2023-08-26T13:57:56.406 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ instance=1 2023-08-26T13:57:56.406 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ daemon_asok_file cluster1:1 cluster1 2023-08-26T13:57:56.406 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ local local_cluster=cluster1:1 2023-08-26T13:57:56.406 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ local cluster=cluster1 2023-08-26T13:57:56.406 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ local instance 2023-08-26T13:57:56.407 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2023-08-26T13:57:56.407 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ local val=cluster1:1 2023-08-26T13:57:56.407 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ local cluster_var_name=local_cluster 2023-08-26T13:57:56.407 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ local instance_var_name=instance 2023-08-26T13:57:56.407 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ cluster=cluster1 2023-08-26T13:57:56.407 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ instance=1 2023-08-26T13:57:56.408 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ '[' 1 = cluster1:1 ']' 2023-08-26T13:57:56.408 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ eval local_cluster=cluster1 2023-08-26T13:57:56.408 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++++ local_cluster=cluster1 2023-08-26T13:57:56.408 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ eval instance=1 2023-08-26T13:57:56.408 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++++ instance=1 2023-08-26T13:57:56.408 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2023-08-26T13:57:56.422 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-08-26T13:57:56.422 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-08-26T13:57:56.422 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-08-26T13:57:56.422 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_40 2023-08-26T13:57:56.537 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:Invalid command: unused arguments: ['mirror/image_40'] 2023-08-26T13:57:56.537 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:rbd mirror status : get status for rbd mirror 2023-08-26T13:57:56.538 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:admin_socket: invalid command 2023-08-26T13:57:56.541 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-08-26T13:57:56.541 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/image_40 2023-08-26T13:57:56.541 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local cluster=cluster1:2 2023-08-26T13:57:56.541 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ shift 2023-08-26T13:57:56.541 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local instance 2023-08-26T13:57:56.542 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ set_cluster_instance cluster1:2 cluster instance 2023-08-26T13:57:56.542 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local val=cluster1:2 2023-08-26T13:57:56.542 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local cluster_var_name=cluster 2023-08-26T13:57:56.542 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local instance_var_name=instance 2023-08-26T13:57:56.542 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ cluster=cluster1 2023-08-26T13:57:56.543 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ instance=2 2023-08-26T13:57:56.543 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ '[' 2 = cluster1:2 ']' 2023-08-26T13:57:56.543 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ eval cluster=cluster1 2023-08-26T13:57:56.543 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ cluster=cluster1 2023-08-26T13:57:56.543 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ eval instance=2 2023-08-26T13:57:56.544 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ instance=2 2023-08-26T13:57:56.544 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ daemon_asok_file cluster1:2 cluster1 2023-08-26T13:57:56.544 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ local local_cluster=cluster1:2 2023-08-26T13:57:56.544 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ local cluster=cluster1 2023-08-26T13:57:56.544 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ local instance 2023-08-26T13:57:56.545 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2023-08-26T13:57:56.545 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ local val=cluster1:2 2023-08-26T13:57:56.545 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ local cluster_var_name=local_cluster 2023-08-26T13:57:56.545 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ local instance_var_name=instance 2023-08-26T13:57:56.545 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ cluster=cluster1 2023-08-26T13:57:56.545 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ instance=2 2023-08-26T13:57:56.546 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ '[' 2 = cluster1:2 ']' 2023-08-26T13:57:56.546 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ eval local_cluster=cluster1 2023-08-26T13:57:56.546 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++++ local_cluster=cluster1 2023-08-26T13:57:56.546 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ eval instance=2 2023-08-26T13:57:56.546 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++++ instance=2 2023-08-26T13:57:56.547 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2023-08-26T13:57:56.559 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-08-26T13:57:56.559 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-08-26T13:57:56.559 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-08-26T13:57:56.560 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/image_40 2023-08-26T13:57:56.656 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:Invalid command: unused arguments: ['mirror/image_40'] 2023-08-26T13:57:56.656 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:rbd mirror status : get status for rbd mirror 2023-08-26T13:57:56.656 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:admin_socket: invalid command 2023-08-26T13:57:56.659 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-08-26T13:57:56.659 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ admin_daemon cluster1:3 rbd mirror status mirror/image_40 2023-08-26T13:57:56.660 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local cluster=cluster1:3 2023-08-26T13:57:56.660 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ shift 2023-08-26T13:57:56.660 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local instance 2023-08-26T13:57:56.660 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ set_cluster_instance cluster1:3 cluster instance 2023-08-26T13:57:56.660 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local val=cluster1:3 2023-08-26T13:57:56.661 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local cluster_var_name=cluster 2023-08-26T13:57:56.661 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local instance_var_name=instance 2023-08-26T13:57:56.661 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ cluster=cluster1 2023-08-26T13:57:56.661 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ instance=3 2023-08-26T13:57:56.661 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ '[' 3 = cluster1:3 ']' 2023-08-26T13:57:56.662 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ eval cluster=cluster1 2023-08-26T13:57:56.662 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ cluster=cluster1 2023-08-26T13:57:56.662 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ eval instance=3 2023-08-26T13:57:56.663 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ instance=3 2023-08-26T13:57:56.663 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ daemon_asok_file cluster1:3 cluster1 2023-08-26T13:57:56.663 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ local local_cluster=cluster1:3 2023-08-26T13:57:56.663 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ local cluster=cluster1 2023-08-26T13:57:56.663 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ local instance 2023-08-26T13:57:56.663 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ set_cluster_instance cluster1:3 local_cluster instance 2023-08-26T13:57:56.664 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ local val=cluster1:3 2023-08-26T13:57:56.664 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ local cluster_var_name=local_cluster 2023-08-26T13:57:56.664 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ local instance_var_name=instance 2023-08-26T13:57:56.664 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ cluster=cluster1 2023-08-26T13:57:56.664 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ instance=3 2023-08-26T13:57:56.665 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ '[' 3 = cluster1:3 ']' 2023-08-26T13:57:56.665 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ eval local_cluster=cluster1 2023-08-26T13:57:56.665 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++++ local_cluster=cluster1 2023-08-26T13:57:56.665 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ eval instance=3 2023-08-26T13:57:56.665 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++++ instance=3 2023-08-26T13:57:56.666 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2023-08-26T13:57:56.677 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2023-08-26T13:57:56.677 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2023-08-26T13:57:56.678 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2023-08-26T13:57:56.678 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror status mirror/image_40 2023-08-26T13:57:56.784 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:Invalid command: unused arguments: ['mirror/image_40'] 2023-08-26T13:57:56.785 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:rbd mirror status : get status for rbd mirror 2023-08-26T13:57:56.785 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:admin_socket: invalid command 2023-08-26T13:57:56.785 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-08-26T13:57:56.786 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ sleep 1 2023-08-26T13:57:57.787 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ '[' cluster1 '!=' cluster1 ']' 2023-08-26T13:57:57.787 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ seq 0 3 2023-08-26T13:57:57.790 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-08-26T13:57:57.791 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_40 2023-08-26T13:57:57.791 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local cluster=cluster1:0 2023-08-26T13:57:57.791 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ shift 2023-08-26T13:57:57.791 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local instance 2023-08-26T13:57:57.791 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ set_cluster_instance cluster1:0 cluster instance 2023-08-26T13:57:57.792 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local val=cluster1:0 2023-08-26T13:57:57.792 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local cluster_var_name=cluster 2023-08-26T13:57:57.792 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local instance_var_name=instance 2023-08-26T13:57:57.792 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ cluster=cluster1 2023-08-26T13:57:57.792 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ instance=0 2023-08-26T13:57:57.793 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ '[' 0 = cluster1:0 ']' 2023-08-26T13:57:57.793 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ eval cluster=cluster1 2023-08-26T13:57:57.793 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ cluster=cluster1 2023-08-26T13:57:57.793 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ eval instance=0 2023-08-26T13:57:57.793 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ instance=0 2023-08-26T13:57:57.793 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ daemon_asok_file cluster1:0 cluster1 2023-08-26T13:57:57.794 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ local local_cluster=cluster1:0 2023-08-26T13:57:57.794 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ local cluster=cluster1 2023-08-26T13:57:57.794 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ local instance 2023-08-26T13:57:57.794 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2023-08-26T13:57:57.794 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ local val=cluster1:0 2023-08-26T13:57:57.795 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ local cluster_var_name=local_cluster 2023-08-26T13:57:57.795 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ local instance_var_name=instance 2023-08-26T13:57:57.795 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ cluster=cluster1 2023-08-26T13:57:57.795 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ instance=0 2023-08-26T13:57:57.795 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ '[' 0 = cluster1:0 ']' 2023-08-26T13:57:57.796 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ eval local_cluster=cluster1 2023-08-26T13:57:57.796 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++++ local_cluster=cluster1 2023-08-26T13:57:57.796 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ eval instance=0 2023-08-26T13:57:57.796 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++++ instance=0 2023-08-26T13:57:57.796 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2023-08-26T13:57:57.813 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-08-26T13:57:57.813 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-08-26T13:57:57.813 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-08-26T13:57:57.813 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_40 2023-08-26T13:57:57.927 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:Invalid command: unused arguments: ['mirror/image_40'] 2023-08-26T13:57:57.928 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:rbd mirror status : get status for rbd mirror 2023-08-26T13:57:57.928 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:admin_socket: invalid command 2023-08-26T13:57:57.931 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-08-26T13:57:57.931 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_40 2023-08-26T13:57:57.931 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local cluster=cluster1:1 2023-08-26T13:57:57.931 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ shift 2023-08-26T13:57:57.931 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local instance 2023-08-26T13:57:57.932 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ set_cluster_instance cluster1:1 cluster instance 2023-08-26T13:57:57.932 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local val=cluster1:1 2023-08-26T13:57:57.932 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local cluster_var_name=cluster 2023-08-26T13:57:57.932 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local instance_var_name=instance 2023-08-26T13:57:57.932 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ cluster=cluster1 2023-08-26T13:57:57.933 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ instance=1 2023-08-26T13:57:57.933 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ '[' 1 = cluster1:1 ']' 2023-08-26T13:57:57.933 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ eval cluster=cluster1 2023-08-26T13:57:57.933 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ cluster=cluster1 2023-08-26T13:57:57.933 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ eval instance=1 2023-08-26T13:57:57.934 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ instance=1 2023-08-26T13:57:57.934 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ daemon_asok_file cluster1:1 cluster1 2023-08-26T13:57:57.934 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ local local_cluster=cluster1:1 2023-08-26T13:57:57.934 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ local cluster=cluster1 2023-08-26T13:57:57.935 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ local instance 2023-08-26T13:57:57.935 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2023-08-26T13:57:57.935 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ local val=cluster1:1 2023-08-26T13:57:57.935 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ local cluster_var_name=local_cluster 2023-08-26T13:57:57.935 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ local instance_var_name=instance 2023-08-26T13:57:57.936 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ cluster=cluster1 2023-08-26T13:57:57.936 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ instance=1 2023-08-26T13:57:57.936 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ '[' 1 = cluster1:1 ']' 2023-08-26T13:57:57.936 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ eval local_cluster=cluster1 2023-08-26T13:57:57.936 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++++ local_cluster=cluster1 2023-08-26T13:57:57.937 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ eval instance=1 2023-08-26T13:57:57.937 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++++ instance=1 2023-08-26T13:57:57.937 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2023-08-26T13:57:57.949 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-08-26T13:57:57.949 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-08-26T13:57:57.950 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-08-26T13:57:57.950 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_40 2023-08-26T13:57:58.053 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:Invalid command: unused arguments: ['mirror/image_40'] 2023-08-26T13:57:58.053 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:rbd mirror status : get status for rbd mirror 2023-08-26T13:57:58.054 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:admin_socket: invalid command 2023-08-26T13:57:58.056 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-08-26T13:57:58.057 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/image_40 2023-08-26T13:57:58.057 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local cluster=cluster1:2 2023-08-26T13:57:58.057 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ shift 2023-08-26T13:57:58.057 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local instance 2023-08-26T13:57:58.058 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ set_cluster_instance cluster1:2 cluster instance 2023-08-26T13:57:58.058 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local val=cluster1:2 2023-08-26T13:57:58.058 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local cluster_var_name=cluster 2023-08-26T13:57:58.058 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local instance_var_name=instance 2023-08-26T13:57:58.059 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ cluster=cluster1 2023-08-26T13:57:58.059 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ instance=2 2023-08-26T13:57:58.059 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ '[' 2 = cluster1:2 ']' 2023-08-26T13:57:58.059 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ eval cluster=cluster1 2023-08-26T13:57:58.059 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ cluster=cluster1 2023-08-26T13:57:58.060 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ eval instance=2 2023-08-26T13:57:58.060 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ instance=2 2023-08-26T13:57:58.060 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ daemon_asok_file cluster1:2 cluster1 2023-08-26T13:57:58.060 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ local local_cluster=cluster1:2 2023-08-26T13:57:58.060 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ local cluster=cluster1 2023-08-26T13:57:58.061 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ local instance 2023-08-26T13:57:58.061 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2023-08-26T13:57:58.061 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ local val=cluster1:2 2023-08-26T13:57:58.061 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ local cluster_var_name=local_cluster 2023-08-26T13:57:58.062 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ local instance_var_name=instance 2023-08-26T13:57:58.062 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ cluster=cluster1 2023-08-26T13:57:58.062 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ instance=2 2023-08-26T13:57:58.062 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ '[' 2 = cluster1:2 ']' 2023-08-26T13:57:58.062 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ eval local_cluster=cluster1 2023-08-26T13:57:58.063 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++++ local_cluster=cluster1 2023-08-26T13:57:58.063 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ eval instance=2 2023-08-26T13:57:58.063 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++++ instance=2 2023-08-26T13:57:58.063 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2023-08-26T13:57:58.075 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-08-26T13:57:58.075 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-08-26T13:57:58.075 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-08-26T13:57:58.076 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/image_40 2023-08-26T13:57:58.173 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:Invalid command: unused arguments: ['mirror/image_40'] 2023-08-26T13:57:58.174 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:rbd mirror status : get status for rbd mirror 2023-08-26T13:57:58.174 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:admin_socket: invalid command 2023-08-26T13:57:58.177 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-08-26T13:57:58.177 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ admin_daemon cluster1:3 rbd mirror status mirror/image_40 2023-08-26T13:57:58.177 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local cluster=cluster1:3 2023-08-26T13:57:58.178 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ shift 2023-08-26T13:57:58.178 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local instance 2023-08-26T13:57:58.178 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ set_cluster_instance cluster1:3 cluster instance 2023-08-26T13:57:58.178 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local val=cluster1:3 2023-08-26T13:57:58.178 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local cluster_var_name=cluster 2023-08-26T13:57:58.179 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local instance_var_name=instance 2023-08-26T13:57:58.179 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ cluster=cluster1 2023-08-26T13:57:58.179 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ instance=3 2023-08-26T13:57:58.179 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ '[' 3 = cluster1:3 ']' 2023-08-26T13:57:58.180 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ eval cluster=cluster1 2023-08-26T13:57:58.180 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ cluster=cluster1 2023-08-26T13:57:58.180 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ eval instance=3 2023-08-26T13:57:58.180 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ instance=3 2023-08-26T13:57:58.180 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ daemon_asok_file cluster1:3 cluster1 2023-08-26T13:57:58.180 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ local local_cluster=cluster1:3 2023-08-26T13:57:58.181 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ local cluster=cluster1 2023-08-26T13:57:58.181 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ local instance 2023-08-26T13:57:58.181 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ set_cluster_instance cluster1:3 local_cluster instance 2023-08-26T13:57:58.181 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ local val=cluster1:3 2023-08-26T13:57:58.181 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ local cluster_var_name=local_cluster 2023-08-26T13:57:58.182 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ local instance_var_name=instance 2023-08-26T13:57:58.182 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ cluster=cluster1 2023-08-26T13:57:58.182 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ instance=3 2023-08-26T13:57:58.182 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ '[' 3 = cluster1:3 ']' 2023-08-26T13:57:58.182 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ eval local_cluster=cluster1 2023-08-26T13:57:58.183 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++++ local_cluster=cluster1 2023-08-26T13:57:58.183 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ eval instance=3 2023-08-26T13:57:58.183 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++++ instance=3 2023-08-26T13:57:58.183 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2023-08-26T13:57:58.195 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2023-08-26T13:57:58.196 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2023-08-26T13:57:58.196 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2023-08-26T13:57:58.196 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror status mirror/image_40 2023-08-26T13:57:58.301 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:Invalid command: unused arguments: ['mirror/image_40'] 2023-08-26T13:57:58.301 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:rbd mirror status : get status for rbd mirror 2023-08-26T13:57:58.301 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:admin_socket: invalid command 2023-08-26T13:57:58.302 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-08-26T13:57:58.302 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ sleep 2 2023-08-26T13:58:00.303 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ '[' cluster1 '!=' cluster1 ']' 2023-08-26T13:58:00.304 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ seq 0 3 2023-08-26T13:58:00.307 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-08-26T13:58:00.307 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_40 2023-08-26T13:58:00.307 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local cluster=cluster1:0 2023-08-26T13:58:00.308 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ shift 2023-08-26T13:58:00.308 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local instance 2023-08-26T13:58:00.308 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ set_cluster_instance cluster1:0 cluster instance 2023-08-26T13:58:00.308 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local val=cluster1:0 2023-08-26T13:58:00.309 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local cluster_var_name=cluster 2023-08-26T13:58:00.309 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local instance_var_name=instance 2023-08-26T13:58:00.309 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ cluster=cluster1 2023-08-26T13:58:00.309 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ instance=0 2023-08-26T13:58:00.309 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ '[' 0 = cluster1:0 ']' 2023-08-26T13:58:00.310 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ eval cluster=cluster1 2023-08-26T13:58:00.310 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ cluster=cluster1 2023-08-26T13:58:00.310 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ eval instance=0 2023-08-26T13:58:00.310 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ instance=0 2023-08-26T13:58:00.310 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ daemon_asok_file cluster1:0 cluster1 2023-08-26T13:58:00.311 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ local local_cluster=cluster1:0 2023-08-26T13:58:00.311 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ local cluster=cluster1 2023-08-26T13:58:00.311 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ local instance 2023-08-26T13:58:00.311 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2023-08-26T13:58:00.311 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ local val=cluster1:0 2023-08-26T13:58:00.312 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ local cluster_var_name=local_cluster 2023-08-26T13:58:00.312 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ local instance_var_name=instance 2023-08-26T13:58:00.312 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ cluster=cluster1 2023-08-26T13:58:00.312 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ instance=0 2023-08-26T13:58:00.312 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ '[' 0 = cluster1:0 ']' 2023-08-26T13:58:00.313 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ eval local_cluster=cluster1 2023-08-26T13:58:00.313 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++++ local_cluster=cluster1 2023-08-26T13:58:00.313 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ eval instance=0 2023-08-26T13:58:00.313 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++++ instance=0 2023-08-26T13:58:00.314 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2023-08-26T13:58:00.328 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-08-26T13:58:00.329 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-08-26T13:58:00.329 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-08-26T13:58:00.329 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_40 2023-08-26T13:58:00.439 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:Invalid command: unused arguments: ['mirror/image_40'] 2023-08-26T13:58:00.439 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:rbd mirror status : get status for rbd mirror 2023-08-26T13:58:00.440 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:admin_socket: invalid command 2023-08-26T13:58:00.442 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-08-26T13:58:00.443 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_40 2023-08-26T13:58:00.443 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local cluster=cluster1:1 2023-08-26T13:58:00.443 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ shift 2023-08-26T13:58:00.443 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local instance 2023-08-26T13:58:00.443 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ set_cluster_instance cluster1:1 cluster instance 2023-08-26T13:58:00.444 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local val=cluster1:1 2023-08-26T13:58:00.444 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local cluster_var_name=cluster 2023-08-26T13:58:00.444 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local instance_var_name=instance 2023-08-26T13:58:00.444 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ cluster=cluster1 2023-08-26T13:58:00.444 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ instance=1 2023-08-26T13:58:00.445 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ '[' 1 = cluster1:1 ']' 2023-08-26T13:58:00.445 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ eval cluster=cluster1 2023-08-26T13:58:00.445 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ cluster=cluster1 2023-08-26T13:58:00.445 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ eval instance=1 2023-08-26T13:58:00.445 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ instance=1 2023-08-26T13:58:00.446 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ daemon_asok_file cluster1:1 cluster1 2023-08-26T13:58:00.446 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ local local_cluster=cluster1:1 2023-08-26T13:58:00.446 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ local cluster=cluster1 2023-08-26T13:58:00.446 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ local instance 2023-08-26T13:58:00.446 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2023-08-26T13:58:00.447 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ local val=cluster1:1 2023-08-26T13:58:00.447 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ local cluster_var_name=local_cluster 2023-08-26T13:58:00.447 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ local instance_var_name=instance 2023-08-26T13:58:00.447 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ cluster=cluster1 2023-08-26T13:58:00.447 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ instance=1 2023-08-26T13:58:00.448 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ '[' 1 = cluster1:1 ']' 2023-08-26T13:58:00.448 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ eval local_cluster=cluster1 2023-08-26T13:58:00.448 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++++ local_cluster=cluster1 2023-08-26T13:58:00.448 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ eval instance=1 2023-08-26T13:58:00.448 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++++ instance=1 2023-08-26T13:58:00.449 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2023-08-26T13:58:00.461 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-08-26T13:58:00.461 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-08-26T13:58:00.461 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-08-26T13:58:00.461 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_40 2023-08-26T13:58:00.565 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:Invalid command: unused arguments: ['mirror/image_40'] 2023-08-26T13:58:00.565 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:rbd mirror status : get status for rbd mirror 2023-08-26T13:58:00.566 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:admin_socket: invalid command 2023-08-26T13:58:00.568 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-08-26T13:58:00.568 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/image_40 2023-08-26T13:58:00.569 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local cluster=cluster1:2 2023-08-26T13:58:00.569 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ shift 2023-08-26T13:58:00.569 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local instance 2023-08-26T13:58:00.569 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ set_cluster_instance cluster1:2 cluster instance 2023-08-26T13:58:00.569 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local val=cluster1:2 2023-08-26T13:58:00.570 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local cluster_var_name=cluster 2023-08-26T13:58:00.570 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local instance_var_name=instance 2023-08-26T13:58:00.570 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ cluster=cluster1 2023-08-26T13:58:00.570 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ instance=2 2023-08-26T13:58:00.570 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ '[' 2 = cluster1:2 ']' 2023-08-26T13:58:00.571 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ eval cluster=cluster1 2023-08-26T13:58:00.571 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ cluster=cluster1 2023-08-26T13:58:00.571 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ eval instance=2 2023-08-26T13:58:00.571 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ instance=2 2023-08-26T13:58:00.571 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ daemon_asok_file cluster1:2 cluster1 2023-08-26T13:58:00.572 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ local local_cluster=cluster1:2 2023-08-26T13:58:00.572 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ local cluster=cluster1 2023-08-26T13:58:00.572 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ local instance 2023-08-26T13:58:00.572 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2023-08-26T13:58:00.572 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ local val=cluster1:2 2023-08-26T13:58:00.573 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ local cluster_var_name=local_cluster 2023-08-26T13:58:00.573 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ local instance_var_name=instance 2023-08-26T13:58:00.573 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ cluster=cluster1 2023-08-26T13:58:00.573 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ instance=2 2023-08-26T13:58:00.573 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ '[' 2 = cluster1:2 ']' 2023-08-26T13:58:00.573 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ eval local_cluster=cluster1 2023-08-26T13:58:00.574 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++++ local_cluster=cluster1 2023-08-26T13:58:00.574 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ eval instance=2 2023-08-26T13:58:00.574 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++++ instance=2 2023-08-26T13:58:00.574 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2023-08-26T13:58:00.587 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-08-26T13:58:00.587 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-08-26T13:58:00.587 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-08-26T13:58:00.588 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/image_40 2023-08-26T13:58:00.687 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:Invalid command: unused arguments: ['mirror/image_40'] 2023-08-26T13:58:00.687 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:rbd mirror status : get status for rbd mirror 2023-08-26T13:58:00.687 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:admin_socket: invalid command 2023-08-26T13:58:00.688 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-08-26T13:58:00.688 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ admin_daemon cluster1:3 rbd mirror status mirror/image_40 2023-08-26T13:58:00.688 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local cluster=cluster1:3 2023-08-26T13:58:00.688 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ shift 2023-08-26T13:58:00.688 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local instance 2023-08-26T13:58:00.689 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ set_cluster_instance cluster1:3 cluster instance 2023-08-26T13:58:00.689 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local val=cluster1:3 2023-08-26T13:58:00.689 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local cluster_var_name=cluster 2023-08-26T13:58:00.689 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local instance_var_name=instance 2023-08-26T13:58:00.689 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ cluster=cluster1 2023-08-26T13:58:00.690 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ instance=3 2023-08-26T13:58:00.690 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ '[' 3 = cluster1:3 ']' 2023-08-26T13:58:00.690 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ eval cluster=cluster1 2023-08-26T13:58:00.690 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ cluster=cluster1 2023-08-26T13:58:00.690 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ eval instance=3 2023-08-26T13:58:00.691 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ instance=3 2023-08-26T13:58:00.691 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ daemon_asok_file cluster1:3 cluster1 2023-08-26T13:58:00.691 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ local local_cluster=cluster1:3 2023-08-26T13:58:00.691 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ local cluster=cluster1 2023-08-26T13:58:00.691 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ local instance 2023-08-26T13:58:00.692 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ set_cluster_instance cluster1:3 local_cluster instance 2023-08-26T13:58:00.692 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ local val=cluster1:3 2023-08-26T13:58:00.692 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ local cluster_var_name=local_cluster 2023-08-26T13:58:00.692 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ local instance_var_name=instance 2023-08-26T13:58:00.692 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ cluster=cluster1 2023-08-26T13:58:00.693 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ instance=3 2023-08-26T13:58:00.693 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ '[' 3 = cluster1:3 ']' 2023-08-26T13:58:00.693 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ eval local_cluster=cluster1 2023-08-26T13:58:00.693 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++++ local_cluster=cluster1 2023-08-26T13:58:00.693 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ eval instance=3 2023-08-26T13:58:00.693 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++++ instance=3 2023-08-26T13:58:00.694 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2023-08-26T13:58:00.705 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2023-08-26T13:58:00.706 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2023-08-26T13:58:00.706 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2023-08-26T13:58:00.706 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror status mirror/image_40 2023-08-26T13:58:00.817 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:Invalid command: unused arguments: ['mirror/image_40'] 2023-08-26T13:58:00.817 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:rbd mirror status : get status for rbd mirror 2023-08-26T13:58:00.817 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:admin_socket: invalid command 2023-08-26T13:58:00.818 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-08-26T13:58:00.818 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ sleep 4 2023-08-26T13:58:04.819 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ '[' cluster1 '!=' cluster1 ']' 2023-08-26T13:58:04.820 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ seq 0 3 2023-08-26T13:58:04.823 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-08-26T13:58:04.823 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_40 2023-08-26T13:58:04.823 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local cluster=cluster1:0 2023-08-26T13:58:04.824 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ shift 2023-08-26T13:58:04.824 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local instance 2023-08-26T13:58:04.824 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ set_cluster_instance cluster1:0 cluster instance 2023-08-26T13:58:04.824 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local val=cluster1:0 2023-08-26T13:58:04.824 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local cluster_var_name=cluster 2023-08-26T13:58:04.825 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local instance_var_name=instance 2023-08-26T13:58:04.825 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ cluster=cluster1 2023-08-26T13:58:04.825 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ instance=0 2023-08-26T13:58:04.825 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ '[' 0 = cluster1:0 ']' 2023-08-26T13:58:04.825 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ eval cluster=cluster1 2023-08-26T13:58:04.826 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ cluster=cluster1 2023-08-26T13:58:04.826 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ eval instance=0 2023-08-26T13:58:04.826 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ instance=0 2023-08-26T13:58:04.826 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ daemon_asok_file cluster1:0 cluster1 2023-08-26T13:58:04.826 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ local local_cluster=cluster1:0 2023-08-26T13:58:04.827 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ local cluster=cluster1 2023-08-26T13:58:04.827 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ local instance 2023-08-26T13:58:04.827 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2023-08-26T13:58:04.827 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ local val=cluster1:0 2023-08-26T13:58:04.827 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ local cluster_var_name=local_cluster 2023-08-26T13:58:04.828 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ local instance_var_name=instance 2023-08-26T13:58:04.828 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ cluster=cluster1 2023-08-26T13:58:04.828 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ instance=0 2023-08-26T13:58:04.828 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ '[' 0 = cluster1:0 ']' 2023-08-26T13:58:04.828 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ eval local_cluster=cluster1 2023-08-26T13:58:04.829 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++++ local_cluster=cluster1 2023-08-26T13:58:04.829 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ eval instance=0 2023-08-26T13:58:04.829 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++++ instance=0 2023-08-26T13:58:04.829 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2023-08-26T13:58:04.845 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-08-26T13:58:04.845 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-08-26T13:58:04.845 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-08-26T13:58:04.845 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_40 2023-08-26T13:58:04.959 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:Invalid command: unused arguments: ['mirror/image_40'] 2023-08-26T13:58:04.959 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:rbd mirror status : get status for rbd mirror 2023-08-26T13:58:04.959 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:admin_socket: invalid command 2023-08-26T13:58:04.962 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-08-26T13:58:04.963 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_40 2023-08-26T13:58:04.963 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local cluster=cluster1:1 2023-08-26T13:58:04.963 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ shift 2023-08-26T13:58:04.963 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local instance 2023-08-26T13:58:04.964 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ set_cluster_instance cluster1:1 cluster instance 2023-08-26T13:58:04.964 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local val=cluster1:1 2023-08-26T13:58:04.964 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local cluster_var_name=cluster 2023-08-26T13:58:04.964 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local instance_var_name=instance 2023-08-26T13:58:04.964 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ cluster=cluster1 2023-08-26T13:58:04.965 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ instance=1 2023-08-26T13:58:04.965 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ '[' 1 = cluster1:1 ']' 2023-08-26T13:58:04.965 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ eval cluster=cluster1 2023-08-26T13:58:04.965 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ cluster=cluster1 2023-08-26T13:58:04.965 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ eval instance=1 2023-08-26T13:58:04.966 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ instance=1 2023-08-26T13:58:04.966 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ daemon_asok_file cluster1:1 cluster1 2023-08-26T13:58:04.966 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ local local_cluster=cluster1:1 2023-08-26T13:58:04.966 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ local cluster=cluster1 2023-08-26T13:58:04.966 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ local instance 2023-08-26T13:58:04.966 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2023-08-26T13:58:04.967 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ local val=cluster1:1 2023-08-26T13:58:04.967 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ local cluster_var_name=local_cluster 2023-08-26T13:58:04.967 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ local instance_var_name=instance 2023-08-26T13:58:04.967 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ cluster=cluster1 2023-08-26T13:58:04.967 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ instance=1 2023-08-26T13:58:04.968 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ '[' 1 = cluster1:1 ']' 2023-08-26T13:58:04.968 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ eval local_cluster=cluster1 2023-08-26T13:58:04.968 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++++ local_cluster=cluster1 2023-08-26T13:58:04.968 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ eval instance=1 2023-08-26T13:58:04.968 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++++ instance=1 2023-08-26T13:58:04.969 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2023-08-26T13:58:04.981 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-08-26T13:58:04.981 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-08-26T13:58:04.981 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-08-26T13:58:04.981 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_40 2023-08-26T13:58:05.084 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:Invalid command: unused arguments: ['mirror/image_40'] 2023-08-26T13:58:05.085 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:rbd mirror status : get status for rbd mirror 2023-08-26T13:58:05.085 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:admin_socket: invalid command 2023-08-26T13:58:05.088 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-08-26T13:58:05.089 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/image_40 2023-08-26T13:58:05.089 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local cluster=cluster1:2 2023-08-26T13:58:05.089 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ shift 2023-08-26T13:58:05.089 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local instance 2023-08-26T13:58:05.090 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ set_cluster_instance cluster1:2 cluster instance 2023-08-26T13:58:05.090 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local val=cluster1:2 2023-08-26T13:58:05.090 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local cluster_var_name=cluster 2023-08-26T13:58:05.090 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local instance_var_name=instance 2023-08-26T13:58:05.090 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ cluster=cluster1 2023-08-26T13:58:05.091 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ instance=2 2023-08-26T13:58:05.091 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ '[' 2 = cluster1:2 ']' 2023-08-26T13:58:05.091 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ eval cluster=cluster1 2023-08-26T13:58:05.091 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ cluster=cluster1 2023-08-26T13:58:05.091 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ eval instance=2 2023-08-26T13:58:05.092 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ instance=2 2023-08-26T13:58:05.092 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ daemon_asok_file cluster1:2 cluster1 2023-08-26T13:58:05.092 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ local local_cluster=cluster1:2 2023-08-26T13:58:05.092 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ local cluster=cluster1 2023-08-26T13:58:05.092 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ local instance 2023-08-26T13:58:05.093 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2023-08-26T13:58:05.093 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ local val=cluster1:2 2023-08-26T13:58:05.093 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ local cluster_var_name=local_cluster 2023-08-26T13:58:05.093 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ local instance_var_name=instance 2023-08-26T13:58:05.093 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ cluster=cluster1 2023-08-26T13:58:05.094 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ instance=2 2023-08-26T13:58:05.094 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ '[' 2 = cluster1:2 ']' 2023-08-26T13:58:05.094 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ eval local_cluster=cluster1 2023-08-26T13:58:05.094 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++++ local_cluster=cluster1 2023-08-26T13:58:05.094 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ eval instance=2 2023-08-26T13:58:05.095 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++++ instance=2 2023-08-26T13:58:05.095 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2023-08-26T13:58:05.106 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-08-26T13:58:05.106 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-08-26T13:58:05.107 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-08-26T13:58:05.107 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/image_40 2023-08-26T13:58:05.204 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:Invalid command: unused arguments: ['mirror/image_40'] 2023-08-26T13:58:05.204 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:rbd mirror status : get status for rbd mirror 2023-08-26T13:58:05.205 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:admin_socket: invalid command 2023-08-26T13:58:05.208 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-08-26T13:58:05.208 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ admin_daemon cluster1:3 rbd mirror status mirror/image_40 2023-08-26T13:58:05.208 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local cluster=cluster1:3 2023-08-26T13:58:05.209 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ shift 2023-08-26T13:58:05.209 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local instance 2023-08-26T13:58:05.209 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ set_cluster_instance cluster1:3 cluster instance 2023-08-26T13:58:05.209 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local val=cluster1:3 2023-08-26T13:58:05.209 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local cluster_var_name=cluster 2023-08-26T13:58:05.210 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local instance_var_name=instance 2023-08-26T13:58:05.210 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ cluster=cluster1 2023-08-26T13:58:05.210 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ instance=3 2023-08-26T13:58:05.210 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ '[' 3 = cluster1:3 ']' 2023-08-26T13:58:05.210 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ eval cluster=cluster1 2023-08-26T13:58:05.210 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ cluster=cluster1 2023-08-26T13:58:05.211 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ eval instance=3 2023-08-26T13:58:05.211 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ instance=3 2023-08-26T13:58:05.211 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ daemon_asok_file cluster1:3 cluster1 2023-08-26T13:58:05.211 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ local local_cluster=cluster1:3 2023-08-26T13:58:05.211 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ local cluster=cluster1 2023-08-26T13:58:05.212 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ local instance 2023-08-26T13:58:05.212 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ set_cluster_instance cluster1:3 local_cluster instance 2023-08-26T13:58:05.212 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ local val=cluster1:3 2023-08-26T13:58:05.212 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ local cluster_var_name=local_cluster 2023-08-26T13:58:05.213 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ local instance_var_name=instance 2023-08-26T13:58:05.213 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ cluster=cluster1 2023-08-26T13:58:05.213 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ instance=3 2023-08-26T13:58:05.213 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ '[' 3 = cluster1:3 ']' 2023-08-26T13:58:05.213 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ eval local_cluster=cluster1 2023-08-26T13:58:05.213 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++++ local_cluster=cluster1 2023-08-26T13:58:05.214 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ eval instance=3 2023-08-26T13:58:05.214 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++++ instance=3 2023-08-26T13:58:05.214 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2023-08-26T13:58:05.227 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2023-08-26T13:58:05.227 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2023-08-26T13:58:05.227 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2023-08-26T13:58:05.227 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror status mirror/image_40 2023-08-26T13:58:05.332 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:Invalid command: unused arguments: ['mirror/image_40'] 2023-08-26T13:58:05.332 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:rbd mirror status : get status for rbd mirror 2023-08-26T13:58:05.332 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:admin_socket: invalid command 2023-08-26T13:58:05.333 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-08-26T13:58:05.333 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ sleep 8 2023-08-26T13:58:13.334 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ '[' cluster1 '!=' cluster1 ']' 2023-08-26T13:58:13.335 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ seq 0 3 2023-08-26T13:58:13.338 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-08-26T13:58:13.338 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_40 2023-08-26T13:58:13.338 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local cluster=cluster1:0 2023-08-26T13:58:13.338 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ shift 2023-08-26T13:58:13.339 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local instance 2023-08-26T13:58:13.339 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ set_cluster_instance cluster1:0 cluster instance 2023-08-26T13:58:13.339 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local val=cluster1:0 2023-08-26T13:58:13.339 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local cluster_var_name=cluster 2023-08-26T13:58:13.339 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local instance_var_name=instance 2023-08-26T13:58:13.340 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ cluster=cluster1 2023-08-26T13:58:13.340 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ instance=0 2023-08-26T13:58:13.340 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ '[' 0 = cluster1:0 ']' 2023-08-26T13:58:13.340 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ eval cluster=cluster1 2023-08-26T13:58:13.340 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ cluster=cluster1 2023-08-26T13:58:13.341 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ eval instance=0 2023-08-26T13:58:13.341 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ instance=0 2023-08-26T13:58:13.341 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ daemon_asok_file cluster1:0 cluster1 2023-08-26T13:58:13.341 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ local local_cluster=cluster1:0 2023-08-26T13:58:13.341 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ local cluster=cluster1 2023-08-26T13:58:13.342 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ local instance 2023-08-26T13:58:13.342 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2023-08-26T13:58:13.342 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ local val=cluster1:0 2023-08-26T13:58:13.342 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ local cluster_var_name=local_cluster 2023-08-26T13:58:13.343 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ local instance_var_name=instance 2023-08-26T13:58:13.343 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ cluster=cluster1 2023-08-26T13:58:13.343 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ instance=0 2023-08-26T13:58:13.343 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ '[' 0 = cluster1:0 ']' 2023-08-26T13:58:13.343 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ eval local_cluster=cluster1 2023-08-26T13:58:13.344 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++++ local_cluster=cluster1 2023-08-26T13:58:13.344 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ eval instance=0 2023-08-26T13:58:13.344 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++++ instance=0 2023-08-26T13:58:13.344 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2023-08-26T13:58:13.360 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-08-26T13:58:13.361 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-08-26T13:58:13.361 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-08-26T13:58:13.361 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_40 2023-08-26T13:58:13.476 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:Invalid command: unused arguments: ['mirror/image_40'] 2023-08-26T13:58:13.477 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:rbd mirror status : get status for rbd mirror 2023-08-26T13:58:13.477 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:admin_socket: invalid command 2023-08-26T13:58:13.479 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-08-26T13:58:13.480 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_40 2023-08-26T13:58:13.480 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local cluster=cluster1:1 2023-08-26T13:58:13.480 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ shift 2023-08-26T13:58:13.480 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local instance 2023-08-26T13:58:13.480 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ set_cluster_instance cluster1:1 cluster instance 2023-08-26T13:58:13.481 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local val=cluster1:1 2023-08-26T13:58:13.481 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local cluster_var_name=cluster 2023-08-26T13:58:13.481 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local instance_var_name=instance 2023-08-26T13:58:13.481 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ cluster=cluster1 2023-08-26T13:58:13.481 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ instance=1 2023-08-26T13:58:13.482 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ '[' 1 = cluster1:1 ']' 2023-08-26T13:58:13.482 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ eval cluster=cluster1 2023-08-26T13:58:13.482 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ cluster=cluster1 2023-08-26T13:58:13.482 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ eval instance=1 2023-08-26T13:58:13.483 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ instance=1 2023-08-26T13:58:13.483 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ daemon_asok_file cluster1:1 cluster1 2023-08-26T13:58:13.483 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ local local_cluster=cluster1:1 2023-08-26T13:58:13.483 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ local cluster=cluster1 2023-08-26T13:58:13.483 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ local instance 2023-08-26T13:58:13.483 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2023-08-26T13:58:13.484 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ local val=cluster1:1 2023-08-26T13:58:13.484 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ local cluster_var_name=local_cluster 2023-08-26T13:58:13.484 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ local instance_var_name=instance 2023-08-26T13:58:13.484 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ cluster=cluster1 2023-08-26T13:58:13.484 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ instance=1 2023-08-26T13:58:13.485 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ '[' 1 = cluster1:1 ']' 2023-08-26T13:58:13.485 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ eval local_cluster=cluster1 2023-08-26T13:58:13.485 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++++ local_cluster=cluster1 2023-08-26T13:58:13.485 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ eval instance=1 2023-08-26T13:58:13.485 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++++ instance=1 2023-08-26T13:58:13.485 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2023-08-26T13:58:13.500 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-08-26T13:58:13.500 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-08-26T13:58:13.500 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-08-26T13:58:13.500 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_40 2023-08-26T13:58:13.603 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:Invalid command: unused arguments: ['mirror/image_40'] 2023-08-26T13:58:13.603 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:rbd mirror status : get status for rbd mirror 2023-08-26T13:58:13.603 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:admin_socket: invalid command 2023-08-26T13:58:13.606 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-08-26T13:58:13.606 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/image_40 2023-08-26T13:58:13.607 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local cluster=cluster1:2 2023-08-26T13:58:13.607 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ shift 2023-08-26T13:58:13.607 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local instance 2023-08-26T13:58:13.607 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ set_cluster_instance cluster1:2 cluster instance 2023-08-26T13:58:13.607 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local val=cluster1:2 2023-08-26T13:58:13.608 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local cluster_var_name=cluster 2023-08-26T13:58:13.608 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local instance_var_name=instance 2023-08-26T13:58:13.608 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ cluster=cluster1 2023-08-26T13:58:13.608 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ instance=2 2023-08-26T13:58:13.608 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ '[' 2 = cluster1:2 ']' 2023-08-26T13:58:13.609 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ eval cluster=cluster1 2023-08-26T13:58:13.609 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ cluster=cluster1 2023-08-26T13:58:13.609 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ eval instance=2 2023-08-26T13:58:13.609 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ instance=2 2023-08-26T13:58:13.609 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ daemon_asok_file cluster1:2 cluster1 2023-08-26T13:58:13.610 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ local local_cluster=cluster1:2 2023-08-26T13:58:13.610 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ local cluster=cluster1 2023-08-26T13:58:13.610 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ local instance 2023-08-26T13:58:13.610 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2023-08-26T13:58:13.610 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ local val=cluster1:2 2023-08-26T13:58:13.610 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ local cluster_var_name=local_cluster 2023-08-26T13:58:13.611 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ local instance_var_name=instance 2023-08-26T13:58:13.611 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ cluster=cluster1 2023-08-26T13:58:13.611 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ instance=2 2023-08-26T13:58:13.611 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ '[' 2 = cluster1:2 ']' 2023-08-26T13:58:13.611 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ eval local_cluster=cluster1 2023-08-26T13:58:13.612 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++++ local_cluster=cluster1 2023-08-26T13:58:13.612 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ eval instance=2 2023-08-26T13:58:13.612 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++++ instance=2 2023-08-26T13:58:13.612 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2023-08-26T13:58:13.625 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-08-26T13:58:13.625 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-08-26T13:58:13.625 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-08-26T13:58:13.625 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/image_40 2023-08-26T13:58:13.722 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:Invalid command: unused arguments: ['mirror/image_40'] 2023-08-26T13:58:13.722 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:rbd mirror status : get status for rbd mirror 2023-08-26T13:58:13.722 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:admin_socket: invalid command 2023-08-26T13:58:13.725 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-08-26T13:58:13.725 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ admin_daemon cluster1:3 rbd mirror status mirror/image_40 2023-08-26T13:58:13.726 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local cluster=cluster1:3 2023-08-26T13:58:13.726 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ shift 2023-08-26T13:58:13.726 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local instance 2023-08-26T13:58:13.726 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ set_cluster_instance cluster1:3 cluster instance 2023-08-26T13:58:13.726 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local val=cluster1:3 2023-08-26T13:58:13.727 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local cluster_var_name=cluster 2023-08-26T13:58:13.727 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local instance_var_name=instance 2023-08-26T13:58:13.727 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ cluster=cluster1 2023-08-26T13:58:13.727 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ instance=3 2023-08-26T13:58:13.727 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ '[' 3 = cluster1:3 ']' 2023-08-26T13:58:13.727 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ eval cluster=cluster1 2023-08-26T13:58:13.728 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ cluster=cluster1 2023-08-26T13:58:13.728 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ eval instance=3 2023-08-26T13:58:13.728 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ instance=3 2023-08-26T13:58:13.728 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ daemon_asok_file cluster1:3 cluster1 2023-08-26T13:58:13.728 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ local local_cluster=cluster1:3 2023-08-26T13:58:13.729 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ local cluster=cluster1 2023-08-26T13:58:13.729 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ local instance 2023-08-26T13:58:13.729 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ set_cluster_instance cluster1:3 local_cluster instance 2023-08-26T13:58:13.729 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ local val=cluster1:3 2023-08-26T13:58:13.729 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ local cluster_var_name=local_cluster 2023-08-26T13:58:13.729 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ local instance_var_name=instance 2023-08-26T13:58:13.730 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ cluster=cluster1 2023-08-26T13:58:13.730 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ instance=3 2023-08-26T13:58:13.730 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ '[' 3 = cluster1:3 ']' 2023-08-26T13:58:13.730 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ eval local_cluster=cluster1 2023-08-26T13:58:13.730 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++++ local_cluster=cluster1 2023-08-26T13:58:13.730 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ eval instance=3 2023-08-26T13:58:13.731 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++++ instance=3 2023-08-26T13:58:13.731 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2023-08-26T13:58:13.745 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2023-08-26T13:58:13.745 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2023-08-26T13:58:13.745 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2023-08-26T13:58:13.745 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror status mirror/image_40 2023-08-26T13:58:13.851 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:Invalid command: unused arguments: ['mirror/image_40'] 2023-08-26T13:58:13.851 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:rbd mirror status : get status for rbd mirror 2023-08-26T13:58:13.851 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:admin_socket: invalid command 2023-08-26T13:58:13.852 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-08-26T13:58:13.852 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ sleep 8 2023-08-26T13:58:21.853 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ '[' cluster1 '!=' cluster1 ']' 2023-08-26T13:58:21.854 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ seq 0 3 2023-08-26T13:58:21.857 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-08-26T13:58:21.857 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_40 2023-08-26T13:58:21.857 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local cluster=cluster1:0 2023-08-26T13:58:21.857 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ shift 2023-08-26T13:58:21.857 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local instance 2023-08-26T13:58:21.858 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ set_cluster_instance cluster1:0 cluster instance 2023-08-26T13:58:21.858 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local val=cluster1:0 2023-08-26T13:58:21.858 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local cluster_var_name=cluster 2023-08-26T13:58:21.858 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local instance_var_name=instance 2023-08-26T13:58:21.858 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ cluster=cluster1 2023-08-26T13:58:21.859 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ instance=0 2023-08-26T13:58:21.859 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ '[' 0 = cluster1:0 ']' 2023-08-26T13:58:21.859 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ eval cluster=cluster1 2023-08-26T13:58:21.859 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ cluster=cluster1 2023-08-26T13:58:21.859 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ eval instance=0 2023-08-26T13:58:21.860 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ instance=0 2023-08-26T13:58:21.860 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ daemon_asok_file cluster1:0 cluster1 2023-08-26T13:58:21.860 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ local local_cluster=cluster1:0 2023-08-26T13:58:21.860 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ local cluster=cluster1 2023-08-26T13:58:21.860 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ local instance 2023-08-26T13:58:21.861 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2023-08-26T13:58:21.861 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ local val=cluster1:0 2023-08-26T13:58:21.861 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ local cluster_var_name=local_cluster 2023-08-26T13:58:21.861 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ local instance_var_name=instance 2023-08-26T13:58:21.861 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ cluster=cluster1 2023-08-26T13:58:21.862 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ instance=0 2023-08-26T13:58:21.862 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ '[' 0 = cluster1:0 ']' 2023-08-26T13:58:21.862 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ eval local_cluster=cluster1 2023-08-26T13:58:21.862 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++++ local_cluster=cluster1 2023-08-26T13:58:21.862 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ eval instance=0 2023-08-26T13:58:21.862 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++++ instance=0 2023-08-26T13:58:21.863 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2023-08-26T13:58:21.878 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-08-26T13:58:21.879 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-08-26T13:58:21.879 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-08-26T13:58:21.879 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_40 2023-08-26T13:58:21.996 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:Invalid command: unused arguments: ['mirror/image_40'] 2023-08-26T13:58:21.996 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:rbd mirror status : get status for rbd mirror 2023-08-26T13:58:21.997 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:admin_socket: invalid command 2023-08-26T13:58:21.999 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-08-26T13:58:22.000 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_40 2023-08-26T13:58:22.000 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local cluster=cluster1:1 2023-08-26T13:58:22.000 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ shift 2023-08-26T13:58:22.000 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local instance 2023-08-26T13:58:22.000 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ set_cluster_instance cluster1:1 cluster instance 2023-08-26T13:58:22.001 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local val=cluster1:1 2023-08-26T13:58:22.001 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local cluster_var_name=cluster 2023-08-26T13:58:22.001 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local instance_var_name=instance 2023-08-26T13:58:22.001 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ cluster=cluster1 2023-08-26T13:58:22.001 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ instance=1 2023-08-26T13:58:22.001 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ '[' 1 = cluster1:1 ']' 2023-08-26T13:58:22.002 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ eval cluster=cluster1 2023-08-26T13:58:22.002 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ cluster=cluster1 2023-08-26T13:58:22.002 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ eval instance=1 2023-08-26T13:58:22.002 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ instance=1 2023-08-26T13:58:22.003 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ daemon_asok_file cluster1:1 cluster1 2023-08-26T13:58:22.003 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ local local_cluster=cluster1:1 2023-08-26T13:58:22.003 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ local cluster=cluster1 2023-08-26T13:58:22.003 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ local instance 2023-08-26T13:58:22.003 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2023-08-26T13:58:22.003 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ local val=cluster1:1 2023-08-26T13:58:22.004 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ local cluster_var_name=local_cluster 2023-08-26T13:58:22.004 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ local instance_var_name=instance 2023-08-26T13:58:22.004 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ cluster=cluster1 2023-08-26T13:58:22.004 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ instance=1 2023-08-26T13:58:22.005 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ '[' 1 = cluster1:1 ']' 2023-08-26T13:58:22.005 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ eval local_cluster=cluster1 2023-08-26T13:58:22.005 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++++ local_cluster=cluster1 2023-08-26T13:58:22.005 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ eval instance=1 2023-08-26T13:58:22.005 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++++ instance=1 2023-08-26T13:58:22.006 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2023-08-26T13:58:22.018 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-08-26T13:58:22.018 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-08-26T13:58:22.019 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-08-26T13:58:22.019 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_40 2023-08-26T13:58:22.124 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:Invalid command: unused arguments: ['mirror/image_40'] 2023-08-26T13:58:22.125 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:rbd mirror status : get status for rbd mirror 2023-08-26T13:58:22.125 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:admin_socket: invalid command 2023-08-26T13:58:22.128 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-08-26T13:58:22.128 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/image_40 2023-08-26T13:58:22.129 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local cluster=cluster1:2 2023-08-26T13:58:22.129 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ shift 2023-08-26T13:58:22.129 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local instance 2023-08-26T13:58:22.129 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ set_cluster_instance cluster1:2 cluster instance 2023-08-26T13:58:22.130 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local val=cluster1:2 2023-08-26T13:58:22.130 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local cluster_var_name=cluster 2023-08-26T13:58:22.130 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local instance_var_name=instance 2023-08-26T13:58:22.130 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ cluster=cluster1 2023-08-26T13:58:22.130 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ instance=2 2023-08-26T13:58:22.131 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ '[' 2 = cluster1:2 ']' 2023-08-26T13:58:22.131 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ eval cluster=cluster1 2023-08-26T13:58:22.131 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ cluster=cluster1 2023-08-26T13:58:22.131 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ eval instance=2 2023-08-26T13:58:22.131 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ instance=2 2023-08-26T13:58:22.132 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ daemon_asok_file cluster1:2 cluster1 2023-08-26T13:58:22.132 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ local local_cluster=cluster1:2 2023-08-26T13:58:22.132 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ local cluster=cluster1 2023-08-26T13:58:22.132 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ local instance 2023-08-26T13:58:22.132 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2023-08-26T13:58:22.132 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ local val=cluster1:2 2023-08-26T13:58:22.133 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ local cluster_var_name=local_cluster 2023-08-26T13:58:22.133 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ local instance_var_name=instance 2023-08-26T13:58:22.133 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ cluster=cluster1 2023-08-26T13:58:22.133 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ instance=2 2023-08-26T13:58:22.133 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ '[' 2 = cluster1:2 ']' 2023-08-26T13:58:22.134 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ eval local_cluster=cluster1 2023-08-26T13:58:22.134 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++++ local_cluster=cluster1 2023-08-26T13:58:22.134 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ eval instance=2 2023-08-26T13:58:22.134 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++++ instance=2 2023-08-26T13:58:22.134 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2023-08-26T13:58:22.146 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-08-26T13:58:22.146 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-08-26T13:58:22.146 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-08-26T13:58:22.147 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/image_40 2023-08-26T13:58:22.260 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ return 0 2023-08-26T13:58:22.261 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ status_result=' "state": "Replaying"' 2023-08-26T13:58:22.261 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ echo ' "state": "Replaying"' 2023-08-26T13:58:22.262 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ grep -i Replaying 2023-08-26T13:58:22.262 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: "state": "Replaying" 2023-08-26T13:58:22.264 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ current_state=started 2023-08-26T13:58:22.264 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ test started = started 2023-08-26T13:58:22.264 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ return 0 2023-08-26T13:58:22.265 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_40 2023-08-26T13:58:22.265 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local local_cluster=cluster1 2023-08-26T13:58:22.265 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster2 2023-08-26T13:58:22.265 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T13:58:22.265 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=image_40 2023-08-26T13:58:22.266 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ '[' snapshot = journal ']' 2023-08-26T13:58:22.266 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ '[' snapshot = snapshot ']' 2023-08-26T13:58:22.266 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_40 2023-08-26T13:58:22.266 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local local_cluster=cluster1 2023-08-26T13:58:22.266 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster2 2023-08-26T13:58:22.267 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T13:58:22.267 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=image_40 2023-08-26T13:58:22.267 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ mkfname cluster2-mirror-image_40.status 2023-08-26T13:58:22.267 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ echo cluster2-mirror-image_40.status 2023-08-26T13:58:22.267 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ sed -e 's|[/ ]|_|g' 2023-08-26T13:58:22.268 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local status_log=/tmp/tmp.TfDn6BcQ3N/cluster2-mirror-image_40.status 2023-08-26T13:58:22.268 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ mkfname cluster1-mirror-image_40.status 2023-08-26T13:58:22.269 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ echo cluster1-mirror-image_40.status 2023-08-26T13:58:22.269 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ sed -e 's|[/ ]|_|g' 2023-08-26T13:58:22.269 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local local_status_log=/tmp/tmp.TfDn6BcQ3N/cluster1-mirror-image_40.status 2023-08-26T13:58:22.269 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ mirror_image_snapshot cluster2 mirror image_40 2023-08-26T13:58:22.270 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster2 2023-08-26T13:58:22.270 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T13:58:22.270 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=image_40 2023-08-26T13:58:22.270 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_40 2023-08-26T13:58:23.303 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:Snapshot ID: 197 2023-08-26T13:58:23.312 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_40 /tmp/tmp.TfDn6BcQ3N/cluster2-mirror-image_40.status 2023-08-26T13:58:23.313 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster2 2023-08-26T13:58:23.313 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T13:58:23.313 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=image_40 2023-08-26T13:58:23.313 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local log=/tmp/tmp.TfDn6BcQ3N/cluster2-mirror-image_40.status 2023-08-26T13:58:23.314 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_40 --format xml 2023-08-26T13:58:23.314 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-08-26T13:58:23.357 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ xmlstarlet sel -t -v //snapshot/id 2023-08-26T13:58:23.359 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local snapshot_id=197 2023-08-26T13:58:23.359 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ true 2023-08-26T13:58:23.360 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-08-26T13:58:23.360 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ sleep 0.2 2023-08-26T13:58:23.561 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_40 /tmp/tmp.TfDn6BcQ3N/cluster1-mirror-image_40.status 2023-08-26T13:58:23.561 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster1 2023-08-26T13:58:23.561 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T13:58:23.561 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=image_40 2023-08-26T13:58:23.562 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local log=/tmp/tmp.TfDn6BcQ3N/cluster1-mirror-image_40.status 2023-08-26T13:58:23.562 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_40 --format xml 2023-08-26T13:58:23.562 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-08-26T13:58:23.609 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ true 2023-08-26T13:58:23.609 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-08-26T13:58:23.610 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:-:1.1: Document is empty 2023-08-26T13:58:23.610 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr: 2023-08-26T13:58:23.610 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:^ 2023-08-26T13:58:23.611 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local primary_snapshot_id= 2023-08-26T13:58:23.611 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ test 197 = '' 2023-08-26T13:58:23.612 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-08-26T13:58:23.612 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ sleep 0.4 2023-08-26T13:58:24.012 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_40 /tmp/tmp.TfDn6BcQ3N/cluster1-mirror-image_40.status 2023-08-26T13:58:24.013 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster1 2023-08-26T13:58:24.013 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T13:58:24.013 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=image_40 2023-08-26T13:58:24.013 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local log=/tmp/tmp.TfDn6BcQ3N/cluster1-mirror-image_40.status 2023-08-26T13:58:24.013 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_40 --format xml 2023-08-26T13:58:24.014 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-08-26T13:58:24.061 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ true 2023-08-26T13:58:24.062 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-08-26T13:58:24.063 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:-:1.1: Document is empty 2023-08-26T13:58:24.063 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr: 2023-08-26T13:58:24.064 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:^ 2023-08-26T13:58:24.064 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local primary_snapshot_id= 2023-08-26T13:58:24.064 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ test 197 = '' 2023-08-26T13:58:24.064 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-08-26T13:58:24.064 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ sleep 0.8 2023-08-26T13:58:24.864 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_40 /tmp/tmp.TfDn6BcQ3N/cluster1-mirror-image_40.status 2023-08-26T13:58:24.865 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster1 2023-08-26T13:58:24.865 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T13:58:24.865 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=image_40 2023-08-26T13:58:24.865 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local log=/tmp/tmp.TfDn6BcQ3N/cluster1-mirror-image_40.status 2023-08-26T13:58:24.865 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_40 --format xml 2023-08-26T13:58:24.866 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-08-26T13:58:24.924 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-08-26T13:58:24.926 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local primary_snapshot_id=197 2023-08-26T13:58:24.927 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ test 197 = 197 2023-08-26T13:58:24.927 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ return 0 2023-08-26T13:58:24.927 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ wait_for_snap_present cluster1 mirror image_40 snap 2023-08-26T13:58:24.927 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster1 2023-08-26T13:58:24.928 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T13:58:24.928 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=image_40 2023-08-26T13:58:24.928 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local snap_name=snap 2023-08-26T13:58:24.928 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local s 2023-08-26T13:58:24.928 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2023-08-26T13:58:24.929 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ sleep 1 2023-08-26T13:58:25.927 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rbd --cluster cluster1 info mirror/image_40@snap 2023-08-26T13:58:25.971 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:rbd image 'image_40': 2023-08-26T13:58:25.972 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: size 128 MiB in 32 objects 2023-08-26T13:58:25.972 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: order 22 (4 MiB objects) 2023-08-26T13:58:25.972 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: snapshot_count: 2 2023-08-26T13:58:25.972 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: id: 12cf3f09dec0 2023-08-26T13:58:25.972 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: block_name_prefix: rbd_data.12cf3f09dec0 2023-08-26T13:58:25.973 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: format: 2 2023-08-26T13:58:25.973 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: features: layering, exclusive-lock, non-primary 2023-08-26T13:58:25.973 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: op_features: 2023-08-26T13:58:25.973 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: flags: 2023-08-26T13:58:25.973 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: create_timestamp: Sat Aug 26 13:49:22 2023 2023-08-26T13:58:25.974 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: access_timestamp: Sat Aug 26 13:49:22 2023 2023-08-26T13:58:25.974 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: modify_timestamp: Sat Aug 26 13:49:22 2023 2023-08-26T13:58:25.974 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: protected: False 2023-08-26T13:58:25.974 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: mirroring state: enabled 2023-08-26T13:58:25.974 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: mirroring mode: snapshot 2023-08-26T13:58:25.975 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: mirroring global id: 5bb319f8-2f9d-4a84-8756-0c66e011e111 2023-08-26T13:58:25.975 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: mirroring primary: false 2023-08-26T13:58:25.976 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ return 0 2023-08-26T13:58:25.976 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ compare_image_snaps mirror image_40 snap 2023-08-26T13:58:25.976 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T13:58:25.977 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=image_40 2023-08-26T13:58:25.977 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local snap_name=snap 2023-08-26T13:58:25.977 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local ret=0 2023-08-26T13:58:25.977 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local rmt_export=/tmp/tmp.TfDn6BcQ3N/cluster2-mirror-image_40.export 2023-08-26T13:58:25.977 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local loc_export=/tmp/tmp.TfDn6BcQ3N/cluster1-mirror-image_40.export 2023-08-26T13:58:25.978 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rm -f /tmp/tmp.TfDn6BcQ3N/cluster2-mirror-image_40.export /tmp/tmp.TfDn6BcQ3N/cluster1-mirror-image_40.export 2023-08-26T13:58:25.978 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rbd --cluster cluster2 -p mirror export image_40@snap /tmp/tmp.TfDn6BcQ3N/cluster2-mirror-image_40.export 2023-08-26T13:58:26.162 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-08-26T13:58:26.167 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rbd --cluster cluster1 -p mirror export image_40@snap /tmp/tmp.TfDn6BcQ3N/cluster1-mirror-image_40.export 2023-08-26T13:58:26.359 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-08-26T13:58:26.365 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ cmp /tmp/tmp.TfDn6BcQ3N/cluster2-mirror-image_40.export /tmp/tmp.TfDn6BcQ3N/cluster1-mirror-image_40.export 2023-08-26T13:58:26.419 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rm -f /tmp/tmp.TfDn6BcQ3N/cluster2-mirror-image_40.export /tmp/tmp.TfDn6BcQ3N/cluster1-mirror-image_40.export 2023-08-26T13:58:26.457 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ return 0 2023-08-26T13:58:26.457 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-08-26T13:58:26.457 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ image=image_41 2023-08-26T13:58:26.457 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ create_snap cluster2 mirror image_41 snap 2023-08-26T13:58:26.458 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster2 2023-08-26T13:58:26.458 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T13:58:26.458 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=image_41 2023-08-26T13:58:26.458 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local snap_name=snap 2023-08-26T13:58:26.459 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rbd --cluster cluster2 -p mirror snap create image_41@snap --debug-rbd=20 --debug-journaler=20 2023-08-26T13:58:28.311 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ wait_for_image_replay_started cluster1 mirror image_41 2023-08-26T13:58:28.311 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster1 2023-08-26T13:58:28.311 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T13:58:28.312 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=image_41 2023-08-26T13:58:28.312 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ wait_for_image_replay_state cluster1 mirror image_41 started 2023-08-26T13:58:28.312 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster1 2023-08-26T13:58:28.312 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T13:58:28.312 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=image_41 2023-08-26T13:58:28.312 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local state=started 2023-08-26T13:58:28.313 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local s 2023-08-26T13:58:28.313 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-08-26T13:58:28.313 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ sleep 1 2023-08-26T13:58:29.313 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ test_image_replay_state cluster1 mirror image_41 started 2023-08-26T13:58:29.313 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster1 2023-08-26T13:58:29.313 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T13:58:29.314 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=image_41 2023-08-26T13:58:29.314 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local test_state=started 2023-08-26T13:58:29.314 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local status_result 2023-08-26T13:58:29.314 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local current_state=stopped 2023-08-26T13:58:29.315 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_41 2023-08-26T13:58:29.315 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ grep -i state 2023-08-26T13:58:29.315 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local cluster_instance=cluster1 2023-08-26T13:58:29.315 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ shift 2023-08-26T13:58:29.315 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local cluster=cluster1 2023-08-26T13:58:29.316 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local instance=cluster1 2023-08-26T13:58:29.316 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local loop_instance 2023-08-26T13:58:29.316 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-08-26T13:58:29.316 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ sleep 0 2023-08-26T13:58:29.320 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ '[' cluster1 '!=' cluster1 ']' 2023-08-26T13:58:29.320 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ seq 0 3 2023-08-26T13:58:29.321 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-08-26T13:58:29.321 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_41 2023-08-26T13:58:29.321 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local cluster=cluster1:0 2023-08-26T13:58:29.321 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ shift 2023-08-26T13:58:29.321 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local instance 2023-08-26T13:58:29.322 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ set_cluster_instance cluster1:0 cluster instance 2023-08-26T13:58:29.322 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local val=cluster1:0 2023-08-26T13:58:29.322 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local cluster_var_name=cluster 2023-08-26T13:58:29.322 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local instance_var_name=instance 2023-08-26T13:58:29.322 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ cluster=cluster1 2023-08-26T13:58:29.323 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ instance=0 2023-08-26T13:58:29.323 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ '[' 0 = cluster1:0 ']' 2023-08-26T13:58:29.323 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ eval cluster=cluster1 2023-08-26T13:58:29.323 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ cluster=cluster1 2023-08-26T13:58:29.324 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ eval instance=0 2023-08-26T13:58:29.324 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ instance=0 2023-08-26T13:58:29.324 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ daemon_asok_file cluster1:0 cluster1 2023-08-26T13:58:29.324 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ local local_cluster=cluster1:0 2023-08-26T13:58:29.324 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ local cluster=cluster1 2023-08-26T13:58:29.325 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ local instance 2023-08-26T13:58:29.325 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2023-08-26T13:58:29.325 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ local val=cluster1:0 2023-08-26T13:58:29.325 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ local cluster_var_name=local_cluster 2023-08-26T13:58:29.325 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ local instance_var_name=instance 2023-08-26T13:58:29.326 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ cluster=cluster1 2023-08-26T13:58:29.326 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ instance=0 2023-08-26T13:58:29.326 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ '[' 0 = cluster1:0 ']' 2023-08-26T13:58:29.326 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ eval local_cluster=cluster1 2023-08-26T13:58:29.326 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++++ local_cluster=cluster1 2023-08-26T13:58:29.327 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ eval instance=0 2023-08-26T13:58:29.327 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++++ instance=0 2023-08-26T13:58:29.327 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2023-08-26T13:58:29.338 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-08-26T13:58:29.338 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-08-26T13:58:29.339 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-08-26T13:58:29.339 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_41 2023-08-26T13:58:29.449 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:Invalid command: unused arguments: ['mirror/image_41'] 2023-08-26T13:58:29.449 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:rbd mirror status : get status for rbd mirror 2023-08-26T13:58:29.449 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:admin_socket: invalid command 2023-08-26T13:58:29.453 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-08-26T13:58:29.453 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_41 2023-08-26T13:58:29.454 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local cluster=cluster1:1 2023-08-26T13:58:29.454 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ shift 2023-08-26T13:58:29.454 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local instance 2023-08-26T13:58:29.454 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ set_cluster_instance cluster1:1 cluster instance 2023-08-26T13:58:29.455 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local val=cluster1:1 2023-08-26T13:58:29.455 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local cluster_var_name=cluster 2023-08-26T13:58:29.456 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local instance_var_name=instance 2023-08-26T13:58:29.456 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ cluster=cluster1 2023-08-26T13:58:29.456 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ instance=1 2023-08-26T13:58:29.456 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ '[' 1 = cluster1:1 ']' 2023-08-26T13:58:29.457 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ eval cluster=cluster1 2023-08-26T13:58:29.457 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ cluster=cluster1 2023-08-26T13:58:29.457 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ eval instance=1 2023-08-26T13:58:29.457 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ instance=1 2023-08-26T13:58:29.457 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ daemon_asok_file cluster1:1 cluster1 2023-08-26T13:58:29.457 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ local local_cluster=cluster1:1 2023-08-26T13:58:29.458 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ local cluster=cluster1 2023-08-26T13:58:29.458 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ local instance 2023-08-26T13:58:29.458 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2023-08-26T13:58:29.458 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ local val=cluster1:1 2023-08-26T13:58:29.458 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ local cluster_var_name=local_cluster 2023-08-26T13:58:29.459 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ local instance_var_name=instance 2023-08-26T13:58:29.459 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ cluster=cluster1 2023-08-26T13:58:29.459 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ instance=1 2023-08-26T13:58:29.459 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ '[' 1 = cluster1:1 ']' 2023-08-26T13:58:29.459 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ eval local_cluster=cluster1 2023-08-26T13:58:29.459 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++++ local_cluster=cluster1 2023-08-26T13:58:29.460 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ eval instance=1 2023-08-26T13:58:29.460 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++++ instance=1 2023-08-26T13:58:29.460 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2023-08-26T13:58:29.470 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-08-26T13:58:29.471 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-08-26T13:58:29.471 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-08-26T13:58:29.471 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_41 2023-08-26T13:58:29.586 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ return 0 2023-08-26T13:58:29.587 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ status_result=' "state": "Replaying"' 2023-08-26T13:58:29.587 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ echo ' "state": "Replaying"' 2023-08-26T13:58:29.587 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ grep -i Replaying 2023-08-26T13:58:29.588 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: "state": "Replaying" 2023-08-26T13:58:29.590 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ current_state=started 2023-08-26T13:58:29.590 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ test started = started 2023-08-26T13:58:29.590 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ return 0 2023-08-26T13:58:29.590 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_41 2023-08-26T13:58:29.591 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local local_cluster=cluster1 2023-08-26T13:58:29.591 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster2 2023-08-26T13:58:29.591 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T13:58:29.591 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=image_41 2023-08-26T13:58:29.591 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ '[' snapshot = journal ']' 2023-08-26T13:58:29.592 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ '[' snapshot = snapshot ']' 2023-08-26T13:58:29.592 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_41 2023-08-26T13:58:29.592 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local local_cluster=cluster1 2023-08-26T13:58:29.592 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster2 2023-08-26T13:58:29.592 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T13:58:29.593 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=image_41 2023-08-26T13:58:29.593 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ mkfname cluster2-mirror-image_41.status 2023-08-26T13:58:29.593 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ echo cluster2-mirror-image_41.status 2023-08-26T13:58:29.593 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ sed -e 's|[/ ]|_|g' 2023-08-26T13:58:29.593 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local status_log=/tmp/tmp.TfDn6BcQ3N/cluster2-mirror-image_41.status 2023-08-26T13:58:29.594 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ mkfname cluster1-mirror-image_41.status 2023-08-26T13:58:29.594 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ echo cluster1-mirror-image_41.status 2023-08-26T13:58:29.595 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ sed -e 's|[/ ]|_|g' 2023-08-26T13:58:29.596 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local local_status_log=/tmp/tmp.TfDn6BcQ3N/cluster1-mirror-image_41.status 2023-08-26T13:58:29.596 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ mirror_image_snapshot cluster2 mirror image_41 2023-08-26T13:58:29.596 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster2 2023-08-26T13:58:29.596 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T13:58:29.596 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=image_41 2023-08-26T13:58:29.597 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_41 2023-08-26T13:58:30.317 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:Snapshot ID: 199 2023-08-26T13:58:30.326 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_41 /tmp/tmp.TfDn6BcQ3N/cluster2-mirror-image_41.status 2023-08-26T13:58:30.327 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster2 2023-08-26T13:58:30.327 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T13:58:30.327 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=image_41 2023-08-26T13:58:30.327 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local log=/tmp/tmp.TfDn6BcQ3N/cluster2-mirror-image_41.status 2023-08-26T13:58:30.327 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_41 --format xml 2023-08-26T13:58:30.328 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-08-26T13:58:30.371 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ xmlstarlet sel -t -v //snapshot/id 2023-08-26T13:58:30.372 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local snapshot_id=199 2023-08-26T13:58:30.373 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ true 2023-08-26T13:58:30.373 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-08-26T13:58:30.373 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ sleep 0.2 2023-08-26T13:58:30.574 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_41 /tmp/tmp.TfDn6BcQ3N/cluster1-mirror-image_41.status 2023-08-26T13:58:30.575 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster1 2023-08-26T13:58:30.575 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T13:58:30.575 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=image_41 2023-08-26T13:58:30.575 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local log=/tmp/tmp.TfDn6BcQ3N/cluster1-mirror-image_41.status 2023-08-26T13:58:30.575 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_41 --format xml 2023-08-26T13:58:30.576 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-08-26T13:58:30.622 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-08-26T13:58:30.624 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local primary_snapshot_id=108 2023-08-26T13:58:30.624 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ test 199 = 108 2023-08-26T13:58:30.624 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-08-26T13:58:30.625 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ sleep 0.4 2023-08-26T13:58:31.025 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_41 /tmp/tmp.TfDn6BcQ3N/cluster1-mirror-image_41.status 2023-08-26T13:58:31.026 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster1 2023-08-26T13:58:31.026 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T13:58:31.026 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=image_41 2023-08-26T13:58:31.026 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local log=/tmp/tmp.TfDn6BcQ3N/cluster1-mirror-image_41.status 2023-08-26T13:58:31.027 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_41 --format xml 2023-08-26T13:58:31.027 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-08-26T13:58:31.070 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-08-26T13:58:31.072 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local primary_snapshot_id=108 2023-08-26T13:58:31.072 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ test 199 = 108 2023-08-26T13:58:31.072 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-08-26T13:58:31.072 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ sleep 0.8 2023-08-26T13:58:31.873 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_41 /tmp/tmp.TfDn6BcQ3N/cluster1-mirror-image_41.status 2023-08-26T13:58:31.874 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster1 2023-08-26T13:58:31.874 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T13:58:31.874 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=image_41 2023-08-26T13:58:31.874 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local log=/tmp/tmp.TfDn6BcQ3N/cluster1-mirror-image_41.status 2023-08-26T13:58:31.874 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_41 --format xml 2023-08-26T13:58:31.875 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-08-26T13:58:31.926 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ true 2023-08-26T13:58:31.926 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-08-26T13:58:31.927 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:-:1.1: Document is empty 2023-08-26T13:58:31.927 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr: 2023-08-26T13:58:31.927 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:^ 2023-08-26T13:58:31.928 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local primary_snapshot_id= 2023-08-26T13:58:31.928 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ test 199 = '' 2023-08-26T13:58:31.929 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-08-26T13:58:31.929 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ sleep 1.6 2023-08-26T13:58:33.529 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_41 /tmp/tmp.TfDn6BcQ3N/cluster1-mirror-image_41.status 2023-08-26T13:58:33.529 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster1 2023-08-26T13:58:33.530 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T13:58:33.530 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=image_41 2023-08-26T13:58:33.530 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local log=/tmp/tmp.TfDn6BcQ3N/cluster1-mirror-image_41.status 2023-08-26T13:58:33.530 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_41 --format xml 2023-08-26T13:58:33.530 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-08-26T13:58:33.580 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-08-26T13:58:33.582 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local primary_snapshot_id=199 2023-08-26T13:58:33.582 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ test 199 = 199 2023-08-26T13:58:33.583 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ return 0 2023-08-26T13:58:33.583 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ wait_for_snap_present cluster1 mirror image_41 snap 2023-08-26T13:58:33.583 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster1 2023-08-26T13:58:33.584 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T13:58:33.584 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=image_41 2023-08-26T13:58:33.584 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local snap_name=snap 2023-08-26T13:58:33.584 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local s 2023-08-26T13:58:33.584 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2023-08-26T13:58:33.585 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ sleep 1 2023-08-26T13:58:34.583 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rbd --cluster cluster1 info mirror/image_41@snap 2023-08-26T13:58:34.629 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:rbd image 'image_41': 2023-08-26T13:58:34.629 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: size 128 MiB in 32 objects 2023-08-26T13:58:34.629 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: order 22 (4 MiB objects) 2023-08-26T13:58:34.629 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: snapshot_count: 2 2023-08-26T13:58:34.630 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: id: 10b7aaa724ce 2023-08-26T13:58:34.630 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: block_name_prefix: rbd_data.10b7aaa724ce 2023-08-26T13:58:34.630 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: format: 2 2023-08-26T13:58:34.630 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: features: layering, exclusive-lock, non-primary 2023-08-26T13:58:34.630 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: op_features: 2023-08-26T13:58:34.630 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: flags: 2023-08-26T13:58:34.631 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: create_timestamp: Sat Aug 26 13:49:32 2023 2023-08-26T13:58:34.631 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: access_timestamp: Sat Aug 26 13:49:32 2023 2023-08-26T13:58:34.631 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: modify_timestamp: Sat Aug 26 13:49:32 2023 2023-08-26T13:58:34.631 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: protected: False 2023-08-26T13:58:34.631 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: mirroring state: enabled 2023-08-26T13:58:34.632 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: mirroring mode: snapshot 2023-08-26T13:58:34.632 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: mirroring global id: bfd70442-dd63-468e-9119-77e3357316c5 2023-08-26T13:58:34.632 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: mirroring primary: false 2023-08-26T13:58:34.633 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ return 0 2023-08-26T13:58:34.634 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ compare_image_snaps mirror image_41 snap 2023-08-26T13:58:34.634 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T13:58:34.634 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=image_41 2023-08-26T13:58:34.634 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local snap_name=snap 2023-08-26T13:58:34.634 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local ret=0 2023-08-26T13:58:34.635 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local rmt_export=/tmp/tmp.TfDn6BcQ3N/cluster2-mirror-image_41.export 2023-08-26T13:58:34.635 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local loc_export=/tmp/tmp.TfDn6BcQ3N/cluster1-mirror-image_41.export 2023-08-26T13:58:34.635 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rm -f /tmp/tmp.TfDn6BcQ3N/cluster2-mirror-image_41.export /tmp/tmp.TfDn6BcQ3N/cluster1-mirror-image_41.export 2023-08-26T13:58:34.635 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rbd --cluster cluster2 -p mirror export image_41@snap /tmp/tmp.TfDn6BcQ3N/cluster2-mirror-image_41.export 2023-08-26T13:58:34.833 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-08-26T13:58:34.840 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rbd --cluster cluster1 -p mirror export image_41@snap /tmp/tmp.TfDn6BcQ3N/cluster1-mirror-image_41.export 2023-08-26T13:58:35.106 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-08-26T13:58:35.113 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ cmp /tmp/tmp.TfDn6BcQ3N/cluster2-mirror-image_41.export /tmp/tmp.TfDn6BcQ3N/cluster1-mirror-image_41.export 2023-08-26T13:58:35.168 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rm -f /tmp/tmp.TfDn6BcQ3N/cluster2-mirror-image_41.export /tmp/tmp.TfDn6BcQ3N/cluster1-mirror-image_41.export 2023-08-26T13:58:35.207 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ return 0 2023-08-26T13:58:35.207 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-08-26T13:58:35.208 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ image=image_42 2023-08-26T13:58:35.208 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ create_snap cluster2 mirror image_42 snap 2023-08-26T13:58:35.208 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster2 2023-08-26T13:58:35.208 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T13:58:35.208 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=image_42 2023-08-26T13:58:35.209 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local snap_name=snap 2023-08-26T13:58:35.209 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rbd --cluster cluster2 -p mirror snap create image_42@snap --debug-rbd=20 --debug-journaler=20 2023-08-26T13:58:37.224 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ wait_for_image_replay_started cluster1 mirror image_42 2023-08-26T13:58:37.224 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster1 2023-08-26T13:58:37.224 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T13:58:37.224 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=image_42 2023-08-26T13:58:37.225 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ wait_for_image_replay_state cluster1 mirror image_42 started 2023-08-26T13:58:37.225 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster1 2023-08-26T13:58:37.225 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T13:58:37.225 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=image_42 2023-08-26T13:58:37.225 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local state=started 2023-08-26T13:58:37.226 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local s 2023-08-26T13:58:37.226 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-08-26T13:58:37.226 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ sleep 1 2023-08-26T13:58:38.226 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ test_image_replay_state cluster1 mirror image_42 started 2023-08-26T13:58:38.226 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster1 2023-08-26T13:58:38.226 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T13:58:38.226 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=image_42 2023-08-26T13:58:38.227 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local test_state=started 2023-08-26T13:58:38.227 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local status_result 2023-08-26T13:58:38.227 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local current_state=stopped 2023-08-26T13:58:38.227 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_42 2023-08-26T13:58:38.227 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ grep -i state 2023-08-26T13:58:38.227 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local cluster_instance=cluster1 2023-08-26T13:58:38.228 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ shift 2023-08-26T13:58:38.228 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local cluster=cluster1 2023-08-26T13:58:38.228 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local instance=cluster1 2023-08-26T13:58:38.228 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local loop_instance 2023-08-26T13:58:38.228 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-08-26T13:58:38.229 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ sleep 0 2023-08-26T13:58:38.230 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ '[' cluster1 '!=' cluster1 ']' 2023-08-26T13:58:38.231 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ seq 0 3 2023-08-26T13:58:38.231 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-08-26T13:58:38.231 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_42 2023-08-26T13:58:38.231 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local cluster=cluster1:0 2023-08-26T13:58:38.231 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ shift 2023-08-26T13:58:38.232 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local instance 2023-08-26T13:58:38.232 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ set_cluster_instance cluster1:0 cluster instance 2023-08-26T13:58:38.232 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local val=cluster1:0 2023-08-26T13:58:38.232 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local cluster_var_name=cluster 2023-08-26T13:58:38.232 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local instance_var_name=instance 2023-08-26T13:58:38.233 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ cluster=cluster1 2023-08-26T13:58:38.233 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ instance=0 2023-08-26T13:58:38.233 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ '[' 0 = cluster1:0 ']' 2023-08-26T13:58:38.233 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ eval cluster=cluster1 2023-08-26T13:58:38.233 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ cluster=cluster1 2023-08-26T13:58:38.234 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ eval instance=0 2023-08-26T13:58:38.234 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ instance=0 2023-08-26T13:58:38.234 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ daemon_asok_file cluster1:0 cluster1 2023-08-26T13:58:38.234 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ local local_cluster=cluster1:0 2023-08-26T13:58:38.234 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ local cluster=cluster1 2023-08-26T13:58:38.234 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ local instance 2023-08-26T13:58:38.236 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2023-08-26T13:58:38.236 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ local val=cluster1:0 2023-08-26T13:58:38.236 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ local cluster_var_name=local_cluster 2023-08-26T13:58:38.236 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ local instance_var_name=instance 2023-08-26T13:58:38.236 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ cluster=cluster1 2023-08-26T13:58:38.237 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ instance=0 2023-08-26T13:58:38.237 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ '[' 0 = cluster1:0 ']' 2023-08-26T13:58:38.237 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ eval local_cluster=cluster1 2023-08-26T13:58:38.237 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++++ local_cluster=cluster1 2023-08-26T13:58:38.237 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ eval instance=0 2023-08-26T13:58:38.238 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++++ instance=0 2023-08-26T13:58:38.238 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2023-08-26T13:58:38.251 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-08-26T13:58:38.252 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-08-26T13:58:38.252 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-08-26T13:58:38.253 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_42 2023-08-26T13:58:38.377 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ return 0 2023-08-26T13:58:38.378 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ status_result=' "state": "Replaying"' 2023-08-26T13:58:38.378 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ echo ' "state": "Replaying"' 2023-08-26T13:58:38.378 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ grep -i Replaying 2023-08-26T13:58:38.379 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: "state": "Replaying" 2023-08-26T13:58:38.380 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ current_state=started 2023-08-26T13:58:38.381 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ test started = started 2023-08-26T13:58:38.381 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ return 0 2023-08-26T13:58:38.381 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_42 2023-08-26T13:58:38.381 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local local_cluster=cluster1 2023-08-26T13:58:38.381 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster2 2023-08-26T13:58:38.382 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T13:58:38.382 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=image_42 2023-08-26T13:58:38.382 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ '[' snapshot = journal ']' 2023-08-26T13:58:38.382 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ '[' snapshot = snapshot ']' 2023-08-26T13:58:38.382 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_42 2023-08-26T13:58:38.383 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local local_cluster=cluster1 2023-08-26T13:58:38.383 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster2 2023-08-26T13:58:38.383 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T13:58:38.383 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=image_42 2023-08-26T13:58:38.383 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ mkfname cluster2-mirror-image_42.status 2023-08-26T13:58:38.384 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ echo cluster2-mirror-image_42.status 2023-08-26T13:58:38.384 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ sed -e 's|[/ ]|_|g' 2023-08-26T13:58:38.384 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local status_log=/tmp/tmp.TfDn6BcQ3N/cluster2-mirror-image_42.status 2023-08-26T13:58:38.384 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ mkfname cluster1-mirror-image_42.status 2023-08-26T13:58:38.385 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ echo cluster1-mirror-image_42.status 2023-08-26T13:58:38.385 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ sed -e 's|[/ ]|_|g' 2023-08-26T13:58:38.386 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local local_status_log=/tmp/tmp.TfDn6BcQ3N/cluster1-mirror-image_42.status 2023-08-26T13:58:38.386 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ mirror_image_snapshot cluster2 mirror image_42 2023-08-26T13:58:38.386 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster2 2023-08-26T13:58:38.386 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T13:58:38.387 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=image_42 2023-08-26T13:58:38.387 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_42 2023-08-26T13:58:38.543 INFO:tasks.rbd_mirror_thrash:kill cluster2.client.mirror.0 2023-08-26T13:58:38.544 INFO:tasks.rbd_mirror.cluster2.client.mirror.0:Sent signal 15 2023-08-26T13:58:38.544 INFO:tasks.rbd_mirror_thrash:waiting for 6 secs before reviving daemons 2023-08-26T13:58:38.545 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi124.stderr:2023-08-26T13:58:38.543+0000 7f03628d3700 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.0 (PID: 129787) UID: 1000 2023-08-26T13:58:39.223 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:Snapshot ID: 201 2023-08-26T13:58:39.231 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_42 /tmp/tmp.TfDn6BcQ3N/cluster2-mirror-image_42.status 2023-08-26T13:58:39.232 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster2 2023-08-26T13:58:39.232 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T13:58:39.232 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=image_42 2023-08-26T13:58:39.232 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local log=/tmp/tmp.TfDn6BcQ3N/cluster2-mirror-image_42.status 2023-08-26T13:58:39.232 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_42 --format xml 2023-08-26T13:58:39.233 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-08-26T13:58:39.274 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ xmlstarlet sel -t -v //snapshot/id 2023-08-26T13:58:39.276 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local snapshot_id=201 2023-08-26T13:58:39.276 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ true 2023-08-26T13:58:39.276 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-08-26T13:58:39.276 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ sleep 0.2 2023-08-26T13:58:39.477 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_42 /tmp/tmp.TfDn6BcQ3N/cluster1-mirror-image_42.status 2023-08-26T13:58:39.477 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster1 2023-08-26T13:58:39.477 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T13:58:39.478 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=image_42 2023-08-26T13:58:39.478 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local log=/tmp/tmp.TfDn6BcQ3N/cluster1-mirror-image_42.status 2023-08-26T13:58:39.478 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_42 --format xml 2023-08-26T13:58:39.478 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-08-26T13:58:39.524 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-08-26T13:58:39.526 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local primary_snapshot_id=109 2023-08-26T13:58:39.526 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ test 201 = 109 2023-08-26T13:58:39.527 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-08-26T13:58:39.527 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ sleep 0.4 2023-08-26T13:58:39.927 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_42 /tmp/tmp.TfDn6BcQ3N/cluster1-mirror-image_42.status 2023-08-26T13:58:39.927 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster1 2023-08-26T13:58:39.927 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T13:58:39.928 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=image_42 2023-08-26T13:58:39.928 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local log=/tmp/tmp.TfDn6BcQ3N/cluster1-mirror-image_42.status 2023-08-26T13:58:39.928 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_42 --format xml 2023-08-26T13:58:39.928 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-08-26T13:58:39.974 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ true 2023-08-26T13:58:39.974 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-08-26T13:58:39.975 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:-:1.1: Document is empty 2023-08-26T13:58:39.976 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr: 2023-08-26T13:58:39.976 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:^ 2023-08-26T13:58:39.977 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local primary_snapshot_id= 2023-08-26T13:58:39.977 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ test 201 = '' 2023-08-26T13:58:39.977 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-08-26T13:58:39.977 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ sleep 0.8 2023-08-26T13:58:40.777 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_42 /tmp/tmp.TfDn6BcQ3N/cluster1-mirror-image_42.status 2023-08-26T13:58:40.778 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster1 2023-08-26T13:58:40.778 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T13:58:40.778 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=image_42 2023-08-26T13:58:40.778 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local log=/tmp/tmp.TfDn6BcQ3N/cluster1-mirror-image_42.status 2023-08-26T13:58:40.778 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_42 --format xml 2023-08-26T13:58:40.779 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-08-26T13:58:40.883 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-08-26T13:58:40.885 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local primary_snapshot_id=201 2023-08-26T13:58:40.885 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ test 201 = 201 2023-08-26T13:58:40.886 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ return 0 2023-08-26T13:58:40.886 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ wait_for_snap_present cluster1 mirror image_42 snap 2023-08-26T13:58:40.886 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster1 2023-08-26T13:58:40.886 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T13:58:40.886 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=image_42 2023-08-26T13:58:40.887 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local snap_name=snap 2023-08-26T13:58:40.887 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local s 2023-08-26T13:58:40.887 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2023-08-26T13:58:40.887 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ sleep 1 2023-08-26T13:58:41.886 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rbd --cluster cluster1 info mirror/image_42@snap 2023-08-26T13:58:41.932 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:rbd image 'image_42': 2023-08-26T13:58:41.932 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: size 128 MiB in 32 objects 2023-08-26T13:58:41.933 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: order 22 (4 MiB objects) 2023-08-26T13:58:41.933 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: snapshot_count: 2 2023-08-26T13:58:41.933 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: id: 122b4fc8cb0c 2023-08-26T13:58:41.933 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: block_name_prefix: rbd_data.122b4fc8cb0c 2023-08-26T13:58:41.933 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: format: 2 2023-08-26T13:58:41.934 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: features: layering, exclusive-lock, non-primary 2023-08-26T13:58:41.934 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: op_features: 2023-08-26T13:58:41.934 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: flags: 2023-08-26T13:58:41.934 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: create_timestamp: Sat Aug 26 13:49:45 2023 2023-08-26T13:58:41.934 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: access_timestamp: Sat Aug 26 13:49:45 2023 2023-08-26T13:58:41.934 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: modify_timestamp: Sat Aug 26 13:49:45 2023 2023-08-26T13:58:41.935 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: protected: False 2023-08-26T13:58:41.935 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: mirroring state: enabled 2023-08-26T13:58:41.935 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: mirroring mode: snapshot 2023-08-26T13:58:41.935 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: mirroring global id: 60b957ec-f068-4fc2-9c4b-457a79a9a72f 2023-08-26T13:58:41.935 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: mirroring primary: false 2023-08-26T13:58:41.937 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ return 0 2023-08-26T13:58:41.937 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ compare_image_snaps mirror image_42 snap 2023-08-26T13:58:41.937 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T13:58:41.937 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=image_42 2023-08-26T13:58:41.938 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local snap_name=snap 2023-08-26T13:58:41.938 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local ret=0 2023-08-26T13:58:41.938 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local rmt_export=/tmp/tmp.TfDn6BcQ3N/cluster2-mirror-image_42.export 2023-08-26T13:58:41.938 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local loc_export=/tmp/tmp.TfDn6BcQ3N/cluster1-mirror-image_42.export 2023-08-26T13:58:41.938 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rm -f /tmp/tmp.TfDn6BcQ3N/cluster2-mirror-image_42.export /tmp/tmp.TfDn6BcQ3N/cluster1-mirror-image_42.export 2023-08-26T13:58:41.939 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rbd --cluster cluster2 -p mirror export image_42@snap /tmp/tmp.TfDn6BcQ3N/cluster2-mirror-image_42.export 2023-08-26T13:58:42.116 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-08-26T13:58:42.121 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rbd --cluster cluster1 -p mirror export image_42@snap /tmp/tmp.TfDn6BcQ3N/cluster1-mirror-image_42.export 2023-08-26T13:58:42.357 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-08-26T13:58:42.364 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ cmp /tmp/tmp.TfDn6BcQ3N/cluster2-mirror-image_42.export /tmp/tmp.TfDn6BcQ3N/cluster1-mirror-image_42.export 2023-08-26T13:58:42.419 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rm -f /tmp/tmp.TfDn6BcQ3N/cluster2-mirror-image_42.export /tmp/tmp.TfDn6BcQ3N/cluster1-mirror-image_42.export 2023-08-26T13:58:42.456 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ return 0 2023-08-26T13:58:42.456 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-08-26T13:58:42.456 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ image=image_43 2023-08-26T13:58:42.456 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ create_snap cluster2 mirror image_43 snap 2023-08-26T13:58:42.457 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster2 2023-08-26T13:58:42.457 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T13:58:42.457 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=image_43 2023-08-26T13:58:42.457 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local snap_name=snap 2023-08-26T13:58:42.457 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rbd --cluster cluster2 -p mirror snap create image_43@snap --debug-rbd=20 --debug-journaler=20 2023-08-26T13:58:44.303 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ wait_for_image_replay_started cluster1 mirror image_43 2023-08-26T13:58:44.304 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster1 2023-08-26T13:58:44.304 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T13:58:44.304 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=image_43 2023-08-26T13:58:44.304 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ wait_for_image_replay_state cluster1 mirror image_43 started 2023-08-26T13:58:44.305 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster1 2023-08-26T13:58:44.305 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T13:58:44.305 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=image_43 2023-08-26T13:58:44.305 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local state=started 2023-08-26T13:58:44.306 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local s 2023-08-26T13:58:44.306 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-08-26T13:58:44.306 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ sleep 1 2023-08-26T13:58:44.544 INFO:tasks.rbd_mirror_thrash:waiting for cluster2.client.mirror.0 2023-08-26T13:58:44.545 INFO:teuthology.orchestra.run:waiting for 600 2023-08-26T13:58:44.545 INFO:tasks.rbd_mirror_thrash:reviving cluster2.client.mirror.0 2023-08-26T13:58:44.545 INFO:tasks.rbd_mirror.cluster2.client.mirror.0:Restarting daemon 2023-08-26T13:58:44.546 DEBUG:teuthology.orchestra.run.smithi124:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.0 2023-08-26T13:58:44.548 INFO:tasks.rbd_mirror.cluster2.client.mirror.0:Started 2023-08-26T13:58:44.549 INFO:tasks.rbd_mirror_thrash:waiting for 104 secs before thrashing 2023-08-26T13:58:45.304 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ test_image_replay_state cluster1 mirror image_43 started 2023-08-26T13:58:45.305 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster1 2023-08-26T13:58:45.305 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T13:58:45.305 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=image_43 2023-08-26T13:58:45.305 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local test_state=started 2023-08-26T13:58:45.305 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local status_result 2023-08-26T13:58:45.306 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local current_state=stopped 2023-08-26T13:58:45.307 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_43 2023-08-26T13:58:45.307 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local cluster_instance=cluster1 2023-08-26T13:58:45.307 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ grep -i state 2023-08-26T13:58:45.307 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ shift 2023-08-26T13:58:45.307 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local cluster=cluster1 2023-08-26T13:58:45.308 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local instance=cluster1 2023-08-26T13:58:45.308 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local loop_instance 2023-08-26T13:58:45.308 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-08-26T13:58:45.308 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ sleep 0 2023-08-26T13:58:45.308 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ '[' cluster1 '!=' cluster1 ']' 2023-08-26T13:58:45.309 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ seq 0 3 2023-08-26T13:58:45.310 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-08-26T13:58:45.311 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_43 2023-08-26T13:58:45.311 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local cluster=cluster1:0 2023-08-26T13:58:45.311 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ shift 2023-08-26T13:58:45.311 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local instance 2023-08-26T13:58:45.311 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ set_cluster_instance cluster1:0 cluster instance 2023-08-26T13:58:45.312 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local val=cluster1:0 2023-08-26T13:58:45.312 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local cluster_var_name=cluster 2023-08-26T13:58:45.312 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local instance_var_name=instance 2023-08-26T13:58:45.312 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ cluster=cluster1 2023-08-26T13:58:45.312 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ instance=0 2023-08-26T13:58:45.313 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ '[' 0 = cluster1:0 ']' 2023-08-26T13:58:45.313 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ eval cluster=cluster1 2023-08-26T13:58:45.313 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ cluster=cluster1 2023-08-26T13:58:45.313 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ eval instance=0 2023-08-26T13:58:45.313 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ instance=0 2023-08-26T13:58:45.313 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ daemon_asok_file cluster1:0 cluster1 2023-08-26T13:58:45.314 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ local local_cluster=cluster1:0 2023-08-26T13:58:45.315 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ local cluster=cluster1 2023-08-26T13:58:45.315 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ local instance 2023-08-26T13:58:45.315 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2023-08-26T13:58:45.316 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ local val=cluster1:0 2023-08-26T13:58:45.316 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ local cluster_var_name=local_cluster 2023-08-26T13:58:45.316 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ local instance_var_name=instance 2023-08-26T13:58:45.316 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ cluster=cluster1 2023-08-26T13:58:45.316 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ instance=0 2023-08-26T13:58:45.317 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ '[' 0 = cluster1:0 ']' 2023-08-26T13:58:45.317 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ eval local_cluster=cluster1 2023-08-26T13:58:45.317 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++++ local_cluster=cluster1 2023-08-26T13:58:45.317 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ eval instance=0 2023-08-26T13:58:45.317 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++++ instance=0 2023-08-26T13:58:45.317 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2023-08-26T13:58:45.331 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-08-26T13:58:45.331 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-08-26T13:58:45.332 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-08-26T13:58:45.332 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_43 2023-08-26T13:58:45.440 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:Invalid command: unused arguments: ['mirror/image_43'] 2023-08-26T13:58:45.441 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:rbd mirror status : get status for rbd mirror 2023-08-26T13:58:45.441 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:admin_socket: invalid command 2023-08-26T13:58:45.444 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-08-26T13:58:45.444 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_43 2023-08-26T13:58:45.444 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local cluster=cluster1:1 2023-08-26T13:58:45.445 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ shift 2023-08-26T13:58:45.445 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local instance 2023-08-26T13:58:45.445 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ set_cluster_instance cluster1:1 cluster instance 2023-08-26T13:58:45.445 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local val=cluster1:1 2023-08-26T13:58:45.445 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local cluster_var_name=cluster 2023-08-26T13:58:45.446 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local instance_var_name=instance 2023-08-26T13:58:45.446 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ cluster=cluster1 2023-08-26T13:58:45.446 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ instance=1 2023-08-26T13:58:45.446 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ '[' 1 = cluster1:1 ']' 2023-08-26T13:58:45.447 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ eval cluster=cluster1 2023-08-26T13:58:45.447 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ cluster=cluster1 2023-08-26T13:58:45.447 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ eval instance=1 2023-08-26T13:58:45.447 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ instance=1 2023-08-26T13:58:45.447 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ daemon_asok_file cluster1:1 cluster1 2023-08-26T13:58:45.448 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ local local_cluster=cluster1:1 2023-08-26T13:58:45.448 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ local cluster=cluster1 2023-08-26T13:58:45.448 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ local instance 2023-08-26T13:58:45.448 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2023-08-26T13:58:45.448 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ local val=cluster1:1 2023-08-26T13:58:45.449 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ local cluster_var_name=local_cluster 2023-08-26T13:58:45.449 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ local instance_var_name=instance 2023-08-26T13:58:45.449 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ cluster=cluster1 2023-08-26T13:58:45.449 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ instance=1 2023-08-26T13:58:45.449 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ '[' 1 = cluster1:1 ']' 2023-08-26T13:58:45.449 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ eval local_cluster=cluster1 2023-08-26T13:58:45.450 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++++ local_cluster=cluster1 2023-08-26T13:58:45.450 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ eval instance=1 2023-08-26T13:58:45.450 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++++ instance=1 2023-08-26T13:58:45.450 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2023-08-26T13:58:45.462 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-08-26T13:58:45.463 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-08-26T13:58:45.463 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-08-26T13:58:45.463 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_43 2023-08-26T13:58:45.568 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:Invalid command: unused arguments: ['mirror/image_43'] 2023-08-26T13:58:45.568 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:rbd mirror status : get status for rbd mirror 2023-08-26T13:58:45.569 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:admin_socket: invalid command 2023-08-26T13:58:45.571 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-08-26T13:58:45.571 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/image_43 2023-08-26T13:58:45.572 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local cluster=cluster1:2 2023-08-26T13:58:45.572 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ shift 2023-08-26T13:58:45.572 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local instance 2023-08-26T13:58:45.572 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ set_cluster_instance cluster1:2 cluster instance 2023-08-26T13:58:45.572 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local val=cluster1:2 2023-08-26T13:58:45.572 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local cluster_var_name=cluster 2023-08-26T13:58:45.573 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local instance_var_name=instance 2023-08-26T13:58:45.573 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ cluster=cluster1 2023-08-26T13:58:45.573 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ instance=2 2023-08-26T13:58:45.573 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ '[' 2 = cluster1:2 ']' 2023-08-26T13:58:45.573 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ eval cluster=cluster1 2023-08-26T13:58:45.574 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ cluster=cluster1 2023-08-26T13:58:45.574 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ eval instance=2 2023-08-26T13:58:45.574 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ instance=2 2023-08-26T13:58:45.574 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ daemon_asok_file cluster1:2 cluster1 2023-08-26T13:58:45.574 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ local local_cluster=cluster1:2 2023-08-26T13:58:45.574 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ local cluster=cluster1 2023-08-26T13:58:45.575 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ local instance 2023-08-26T13:58:45.575 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2023-08-26T13:58:45.575 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ local val=cluster1:2 2023-08-26T13:58:45.575 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ local cluster_var_name=local_cluster 2023-08-26T13:58:45.575 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ local instance_var_name=instance 2023-08-26T13:58:45.576 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ cluster=cluster1 2023-08-26T13:58:45.576 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ instance=2 2023-08-26T13:58:45.576 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ '[' 2 = cluster1:2 ']' 2023-08-26T13:58:45.576 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ eval local_cluster=cluster1 2023-08-26T13:58:45.576 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++++ local_cluster=cluster1 2023-08-26T13:58:45.576 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ eval instance=2 2023-08-26T13:58:45.577 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++++ instance=2 2023-08-26T13:58:45.577 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2023-08-26T13:58:45.590 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-08-26T13:58:45.590 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-08-26T13:58:45.590 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-08-26T13:58:45.591 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/image_43 2023-08-26T13:58:45.705 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ return 0 2023-08-26T13:58:45.706 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ status_result=' "state": "Replaying"' 2023-08-26T13:58:45.706 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ echo ' "state": "Replaying"' 2023-08-26T13:58:45.707 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ grep -i Replaying 2023-08-26T13:58:45.707 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: "state": "Replaying" 2023-08-26T13:58:45.709 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ current_state=started 2023-08-26T13:58:45.709 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ test started = started 2023-08-26T13:58:45.709 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ return 0 2023-08-26T13:58:45.709 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_43 2023-08-26T13:58:45.710 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local local_cluster=cluster1 2023-08-26T13:58:45.710 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster2 2023-08-26T13:58:45.710 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T13:58:45.710 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=image_43 2023-08-26T13:58:45.710 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ '[' snapshot = journal ']' 2023-08-26T13:58:45.711 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ '[' snapshot = snapshot ']' 2023-08-26T13:58:45.711 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_43 2023-08-26T13:58:45.711 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local local_cluster=cluster1 2023-08-26T13:58:45.711 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster2 2023-08-26T13:58:45.711 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T13:58:45.711 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=image_43 2023-08-26T13:58:45.712 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ mkfname cluster2-mirror-image_43.status 2023-08-26T13:58:45.712 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ echo cluster2-mirror-image_43.status 2023-08-26T13:58:45.712 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ sed -e 's|[/ ]|_|g' 2023-08-26T13:58:45.712 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local status_log=/tmp/tmp.TfDn6BcQ3N/cluster2-mirror-image_43.status 2023-08-26T13:58:45.712 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ mkfname cluster1-mirror-image_43.status 2023-08-26T13:58:45.713 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ echo cluster1-mirror-image_43.status 2023-08-26T13:58:45.714 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ sed -e 's|[/ ]|_|g' 2023-08-26T13:58:45.714 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local local_status_log=/tmp/tmp.TfDn6BcQ3N/cluster1-mirror-image_43.status 2023-08-26T13:58:45.714 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ mirror_image_snapshot cluster2 mirror image_43 2023-08-26T13:58:45.714 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster2 2023-08-26T13:58:45.714 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T13:58:45.714 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=image_43 2023-08-26T13:58:45.715 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_43 2023-08-26T13:58:46.313 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:Snapshot ID: 203 2023-08-26T13:58:46.322 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_43 /tmp/tmp.TfDn6BcQ3N/cluster2-mirror-image_43.status 2023-08-26T13:58:46.322 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster2 2023-08-26T13:58:46.322 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T13:58:46.323 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=image_43 2023-08-26T13:58:46.323 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local log=/tmp/tmp.TfDn6BcQ3N/cluster2-mirror-image_43.status 2023-08-26T13:58:46.323 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_43 --format xml 2023-08-26T13:58:46.323 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-08-26T13:58:46.365 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ xmlstarlet sel -t -v //snapshot/id 2023-08-26T13:58:46.367 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local snapshot_id=203 2023-08-26T13:58:46.367 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ true 2023-08-26T13:58:46.367 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-08-26T13:58:46.367 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ sleep 0.2 2023-08-26T13:58:46.568 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_43 /tmp/tmp.TfDn6BcQ3N/cluster1-mirror-image_43.status 2023-08-26T13:58:46.568 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster1 2023-08-26T13:58:46.568 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T13:58:46.569 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=image_43 2023-08-26T13:58:46.569 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local log=/tmp/tmp.TfDn6BcQ3N/cluster1-mirror-image_43.status 2023-08-26T13:58:46.569 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_43 --format xml 2023-08-26T13:58:46.569 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-08-26T13:58:46.615 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ true 2023-08-26T13:58:46.615 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-08-26T13:58:46.616 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:-:1.1: Document is empty 2023-08-26T13:58:46.616 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr: 2023-08-26T13:58:46.616 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:^ 2023-08-26T13:58:46.617 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local primary_snapshot_id= 2023-08-26T13:58:46.617 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ test 203 = '' 2023-08-26T13:58:46.618 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-08-26T13:58:46.618 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ sleep 0.4 2023-08-26T13:58:47.018 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_43 /tmp/tmp.TfDn6BcQ3N/cluster1-mirror-image_43.status 2023-08-26T13:58:47.018 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster1 2023-08-26T13:58:47.018 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T13:58:47.018 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=image_43 2023-08-26T13:58:47.019 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local log=/tmp/tmp.TfDn6BcQ3N/cluster1-mirror-image_43.status 2023-08-26T13:58:47.019 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_43 --format xml 2023-08-26T13:58:47.019 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-08-26T13:58:47.066 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ true 2023-08-26T13:58:47.067 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-08-26T13:58:47.068 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:-:1.1: Document is empty 2023-08-26T13:58:47.068 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr: 2023-08-26T13:58:47.069 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:^ 2023-08-26T13:58:47.069 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local primary_snapshot_id= 2023-08-26T13:58:47.069 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ test 203 = '' 2023-08-26T13:58:47.069 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-08-26T13:58:47.069 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ sleep 0.8 2023-08-26T13:58:47.870 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_43 /tmp/tmp.TfDn6BcQ3N/cluster1-mirror-image_43.status 2023-08-26T13:58:47.870 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster1 2023-08-26T13:58:47.870 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T13:58:47.871 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=image_43 2023-08-26T13:58:47.871 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local log=/tmp/tmp.TfDn6BcQ3N/cluster1-mirror-image_43.status 2023-08-26T13:58:47.871 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_43 --format xml 2023-08-26T13:58:47.871 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-08-26T13:58:47.916 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-08-26T13:58:47.918 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local primary_snapshot_id=203 2023-08-26T13:58:47.918 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ test 203 = 203 2023-08-26T13:58:47.919 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ return 0 2023-08-26T13:58:47.919 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ wait_for_snap_present cluster1 mirror image_43 snap 2023-08-26T13:58:47.919 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster1 2023-08-26T13:58:47.919 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T13:58:47.920 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=image_43 2023-08-26T13:58:47.920 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local snap_name=snap 2023-08-26T13:58:47.920 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local s 2023-08-26T13:58:47.920 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2023-08-26T13:58:47.920 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ sleep 1 2023-08-26T13:58:48.919 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rbd --cluster cluster1 info mirror/image_43@snap 2023-08-26T13:58:48.964 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:rbd image 'image_43': 2023-08-26T13:58:48.964 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: size 128 MiB in 32 objects 2023-08-26T13:58:48.965 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: order 22 (4 MiB objects) 2023-08-26T13:58:48.965 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: snapshot_count: 2 2023-08-26T13:58:48.965 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: id: 12c0f3380599 2023-08-26T13:58:48.965 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: block_name_prefix: rbd_data.12c0f3380599 2023-08-26T13:58:48.965 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: format: 2 2023-08-26T13:58:48.966 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: features: layering, exclusive-lock, non-primary 2023-08-26T13:58:48.966 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: op_features: 2023-08-26T13:58:48.966 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: flags: 2023-08-26T13:58:48.966 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: create_timestamp: Sat Aug 26 13:49:52 2023 2023-08-26T13:58:48.967 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: access_timestamp: Sat Aug 26 13:49:52 2023 2023-08-26T13:58:48.967 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: modify_timestamp: Sat Aug 26 13:49:52 2023 2023-08-26T13:58:48.967 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: protected: False 2023-08-26T13:58:48.967 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: mirroring state: enabled 2023-08-26T13:58:48.967 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: mirroring mode: snapshot 2023-08-26T13:58:48.968 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: mirroring global id: 93289c2f-b179-4509-b3aa-d0fd920c15e7 2023-08-26T13:58:48.968 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: mirroring primary: false 2023-08-26T13:58:48.969 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ return 0 2023-08-26T13:58:48.969 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ compare_image_snaps mirror image_43 snap 2023-08-26T13:58:48.969 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T13:58:48.970 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=image_43 2023-08-26T13:58:48.970 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local snap_name=snap 2023-08-26T13:58:48.970 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local ret=0 2023-08-26T13:58:48.970 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local rmt_export=/tmp/tmp.TfDn6BcQ3N/cluster2-mirror-image_43.export 2023-08-26T13:58:48.970 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local loc_export=/tmp/tmp.TfDn6BcQ3N/cluster1-mirror-image_43.export 2023-08-26T13:58:48.971 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rm -f /tmp/tmp.TfDn6BcQ3N/cluster2-mirror-image_43.export /tmp/tmp.TfDn6BcQ3N/cluster1-mirror-image_43.export 2023-08-26T13:58:48.971 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rbd --cluster cluster2 -p mirror export image_43@snap /tmp/tmp.TfDn6BcQ3N/cluster2-mirror-image_43.export 2023-08-26T13:58:49.151 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-08-26T13:58:49.156 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rbd --cluster cluster1 -p mirror export image_43@snap /tmp/tmp.TfDn6BcQ3N/cluster1-mirror-image_43.export 2023-08-26T13:58:49.384 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-08-26T13:58:49.392 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ cmp /tmp/tmp.TfDn6BcQ3N/cluster2-mirror-image_43.export /tmp/tmp.TfDn6BcQ3N/cluster1-mirror-image_43.export 2023-08-26T13:58:49.447 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rm -f /tmp/tmp.TfDn6BcQ3N/cluster2-mirror-image_43.export /tmp/tmp.TfDn6BcQ3N/cluster1-mirror-image_43.export 2023-08-26T13:58:49.484 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ return 0 2023-08-26T13:58:49.485 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-08-26T13:58:49.485 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ image=image_44 2023-08-26T13:58:49.485 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ create_snap cluster2 mirror image_44 snap 2023-08-26T13:58:49.485 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster2 2023-08-26T13:58:49.485 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T13:58:49.486 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=image_44 2023-08-26T13:58:49.486 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local snap_name=snap 2023-08-26T13:58:49.486 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rbd --cluster cluster2 -p mirror snap create image_44@snap --debug-rbd=20 --debug-journaler=20 2023-08-26T13:58:51.330 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ wait_for_image_replay_started cluster1 mirror image_44 2023-08-26T13:58:51.331 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster1 2023-08-26T13:58:51.331 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T13:58:51.331 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=image_44 2023-08-26T13:58:51.331 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ wait_for_image_replay_state cluster1 mirror image_44 started 2023-08-26T13:58:51.331 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster1 2023-08-26T13:58:51.331 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T13:58:51.332 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=image_44 2023-08-26T13:58:51.332 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local state=started 2023-08-26T13:58:51.332 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local s 2023-08-26T13:58:51.332 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-08-26T13:58:51.332 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ sleep 1 2023-08-26T13:58:52.331 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ test_image_replay_state cluster1 mirror image_44 started 2023-08-26T13:58:52.332 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster1 2023-08-26T13:58:52.332 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T13:58:52.332 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=image_44 2023-08-26T13:58:52.332 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local test_state=started 2023-08-26T13:58:52.333 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local status_result 2023-08-26T13:58:52.333 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local current_state=stopped 2023-08-26T13:58:52.334 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_44 2023-08-26T13:58:52.334 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ grep -i state 2023-08-26T13:58:52.334 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local cluster_instance=cluster1 2023-08-26T13:58:52.335 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ shift 2023-08-26T13:58:52.335 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local cluster=cluster1 2023-08-26T13:58:52.335 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local instance=cluster1 2023-08-26T13:58:52.335 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local loop_instance 2023-08-26T13:58:52.335 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-08-26T13:58:52.336 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ sleep 0 2023-08-26T13:58:52.336 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ '[' cluster1 '!=' cluster1 ']' 2023-08-26T13:58:52.336 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ seq 0 3 2023-08-26T13:58:52.339 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-08-26T13:58:52.339 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_44 2023-08-26T13:58:52.339 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local cluster=cluster1:0 2023-08-26T13:58:52.339 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ shift 2023-08-26T13:58:52.340 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local instance 2023-08-26T13:58:52.340 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ set_cluster_instance cluster1:0 cluster instance 2023-08-26T13:58:52.340 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local val=cluster1:0 2023-08-26T13:58:52.340 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local cluster_var_name=cluster 2023-08-26T13:58:52.340 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local instance_var_name=instance 2023-08-26T13:58:52.341 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ cluster=cluster1 2023-08-26T13:58:52.341 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ instance=0 2023-08-26T13:58:52.341 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ '[' 0 = cluster1:0 ']' 2023-08-26T13:58:52.341 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ eval cluster=cluster1 2023-08-26T13:58:52.341 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ cluster=cluster1 2023-08-26T13:58:52.342 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ eval instance=0 2023-08-26T13:58:52.342 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ instance=0 2023-08-26T13:58:52.342 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ daemon_asok_file cluster1:0 cluster1 2023-08-26T13:58:52.342 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ local local_cluster=cluster1:0 2023-08-26T13:58:52.342 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ local cluster=cluster1 2023-08-26T13:58:52.343 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ local instance 2023-08-26T13:58:52.343 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2023-08-26T13:58:52.343 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ local val=cluster1:0 2023-08-26T13:58:52.343 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ local cluster_var_name=local_cluster 2023-08-26T13:58:52.343 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ local instance_var_name=instance 2023-08-26T13:58:52.343 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ cluster=cluster1 2023-08-26T13:58:52.344 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ instance=0 2023-08-26T13:58:52.344 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ '[' 0 = cluster1:0 ']' 2023-08-26T13:58:52.344 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ eval local_cluster=cluster1 2023-08-26T13:58:52.344 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++++ local_cluster=cluster1 2023-08-26T13:58:52.344 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ eval instance=0 2023-08-26T13:58:52.345 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++++ instance=0 2023-08-26T13:58:52.345 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2023-08-26T13:58:52.359 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-08-26T13:58:52.359 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-08-26T13:58:52.359 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-08-26T13:58:52.359 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_44 2023-08-26T13:58:52.472 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:Invalid command: unused arguments: ['mirror/image_44'] 2023-08-26T13:58:52.472 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:rbd mirror status : get status for rbd mirror 2023-08-26T13:58:52.473 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:admin_socket: invalid command 2023-08-26T13:58:52.476 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-08-26T13:58:52.477 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_44 2023-08-26T13:58:52.477 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local cluster=cluster1:1 2023-08-26T13:58:52.477 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ shift 2023-08-26T13:58:52.477 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local instance 2023-08-26T13:58:52.478 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ set_cluster_instance cluster1:1 cluster instance 2023-08-26T13:58:52.478 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local val=cluster1:1 2023-08-26T13:58:52.478 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local cluster_var_name=cluster 2023-08-26T13:58:52.478 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local instance_var_name=instance 2023-08-26T13:58:52.478 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ cluster=cluster1 2023-08-26T13:58:52.479 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ instance=1 2023-08-26T13:58:52.479 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ '[' 1 = cluster1:1 ']' 2023-08-26T13:58:52.479 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ eval cluster=cluster1 2023-08-26T13:58:52.479 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ cluster=cluster1 2023-08-26T13:58:52.479 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ eval instance=1 2023-08-26T13:58:52.480 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ instance=1 2023-08-26T13:58:52.480 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ daemon_asok_file cluster1:1 cluster1 2023-08-26T13:58:52.480 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ local local_cluster=cluster1:1 2023-08-26T13:58:52.480 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ local cluster=cluster1 2023-08-26T13:58:52.480 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ local instance 2023-08-26T13:58:52.481 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2023-08-26T13:58:52.481 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ local val=cluster1:1 2023-08-26T13:58:52.481 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ local cluster_var_name=local_cluster 2023-08-26T13:58:52.481 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ local instance_var_name=instance 2023-08-26T13:58:52.481 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ cluster=cluster1 2023-08-26T13:58:52.481 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ instance=1 2023-08-26T13:58:52.482 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ '[' 1 = cluster1:1 ']' 2023-08-26T13:58:52.482 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ eval local_cluster=cluster1 2023-08-26T13:58:52.482 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++++ local_cluster=cluster1 2023-08-26T13:58:52.482 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ eval instance=1 2023-08-26T13:58:52.482 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++++ instance=1 2023-08-26T13:58:52.483 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2023-08-26T13:58:52.494 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-08-26T13:58:52.494 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-08-26T13:58:52.495 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-08-26T13:58:52.495 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_44 2023-08-26T13:58:52.600 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:Invalid command: unused arguments: ['mirror/image_44'] 2023-08-26T13:58:52.601 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:rbd mirror status : get status for rbd mirror 2023-08-26T13:58:52.601 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:admin_socket: invalid command 2023-08-26T13:58:52.604 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-08-26T13:58:52.605 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/image_44 2023-08-26T13:58:52.605 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local cluster=cluster1:2 2023-08-26T13:58:52.605 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ shift 2023-08-26T13:58:52.605 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local instance 2023-08-26T13:58:52.606 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ set_cluster_instance cluster1:2 cluster instance 2023-08-26T13:58:52.606 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local val=cluster1:2 2023-08-26T13:58:52.606 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local cluster_var_name=cluster 2023-08-26T13:58:52.606 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local instance_var_name=instance 2023-08-26T13:58:52.606 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ cluster=cluster1 2023-08-26T13:58:52.607 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ instance=2 2023-08-26T13:58:52.607 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ '[' 2 = cluster1:2 ']' 2023-08-26T13:58:52.607 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ eval cluster=cluster1 2023-08-26T13:58:52.607 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ cluster=cluster1 2023-08-26T13:58:52.607 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ eval instance=2 2023-08-26T13:58:52.608 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ instance=2 2023-08-26T13:58:52.608 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ daemon_asok_file cluster1:2 cluster1 2023-08-26T13:58:52.608 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ local local_cluster=cluster1:2 2023-08-26T13:58:52.608 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ local cluster=cluster1 2023-08-26T13:58:52.608 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ local instance 2023-08-26T13:58:52.609 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2023-08-26T13:58:52.609 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ local val=cluster1:2 2023-08-26T13:58:52.609 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ local cluster_var_name=local_cluster 2023-08-26T13:58:52.609 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ local instance_var_name=instance 2023-08-26T13:58:52.609 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ cluster=cluster1 2023-08-26T13:58:52.609 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ instance=2 2023-08-26T13:58:52.610 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ '[' 2 = cluster1:2 ']' 2023-08-26T13:58:52.610 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ eval local_cluster=cluster1 2023-08-26T13:58:52.610 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++++ local_cluster=cluster1 2023-08-26T13:58:52.610 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ eval instance=2 2023-08-26T13:58:52.610 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++++ instance=2 2023-08-26T13:58:52.610 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2023-08-26T13:58:52.622 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-08-26T13:58:52.623 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-08-26T13:58:52.623 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-08-26T13:58:52.623 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/image_44 2023-08-26T13:58:52.735 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ return 0 2023-08-26T13:58:52.736 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ status_result=' "state": "Replaying"' 2023-08-26T13:58:52.736 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ echo ' "state": "Replaying"' 2023-08-26T13:58:52.736 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ grep -i Replaying 2023-08-26T13:58:52.737 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: "state": "Replaying" 2023-08-26T13:58:52.739 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ current_state=started 2023-08-26T13:58:52.739 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ test started = started 2023-08-26T13:58:52.739 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ return 0 2023-08-26T13:58:52.739 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_44 2023-08-26T13:58:52.739 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local local_cluster=cluster1 2023-08-26T13:58:52.740 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster2 2023-08-26T13:58:52.740 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T13:58:52.740 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=image_44 2023-08-26T13:58:52.740 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ '[' snapshot = journal ']' 2023-08-26T13:58:52.740 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ '[' snapshot = snapshot ']' 2023-08-26T13:58:52.741 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_44 2023-08-26T13:58:52.741 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local local_cluster=cluster1 2023-08-26T13:58:52.741 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster2 2023-08-26T13:58:52.741 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T13:58:52.741 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=image_44 2023-08-26T13:58:52.741 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ mkfname cluster2-mirror-image_44.status 2023-08-26T13:58:52.742 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ echo cluster2-mirror-image_44.status 2023-08-26T13:58:52.742 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ sed -e 's|[/ ]|_|g' 2023-08-26T13:58:52.742 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local status_log=/tmp/tmp.TfDn6BcQ3N/cluster2-mirror-image_44.status 2023-08-26T13:58:52.742 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ mkfname cluster1-mirror-image_44.status 2023-08-26T13:58:52.742 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ echo cluster1-mirror-image_44.status 2023-08-26T13:58:52.743 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ sed -e 's|[/ ]|_|g' 2023-08-26T13:58:52.743 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local local_status_log=/tmp/tmp.TfDn6BcQ3N/cluster1-mirror-image_44.status 2023-08-26T13:58:52.744 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ mirror_image_snapshot cluster2 mirror image_44 2023-08-26T13:58:52.744 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster2 2023-08-26T13:58:52.745 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T13:58:52.745 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=image_44 2023-08-26T13:58:52.745 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_44 2023-08-26T13:58:53.337 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:Snapshot ID: 205 2023-08-26T13:58:53.345 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_44 /tmp/tmp.TfDn6BcQ3N/cluster2-mirror-image_44.status 2023-08-26T13:58:53.345 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster2 2023-08-26T13:58:53.346 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T13:58:53.346 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=image_44 2023-08-26T13:58:53.346 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local log=/tmp/tmp.TfDn6BcQ3N/cluster2-mirror-image_44.status 2023-08-26T13:58:53.346 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_44 --format xml 2023-08-26T13:58:53.346 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-08-26T13:58:53.389 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ xmlstarlet sel -t -v //snapshot/id 2023-08-26T13:58:53.391 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local snapshot_id=205 2023-08-26T13:58:53.392 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ true 2023-08-26T13:58:53.392 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-08-26T13:58:53.393 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ sleep 0.2 2023-08-26T13:58:53.592 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_44 /tmp/tmp.TfDn6BcQ3N/cluster1-mirror-image_44.status 2023-08-26T13:58:53.592 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster1 2023-08-26T13:58:53.593 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T13:58:53.593 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=image_44 2023-08-26T13:58:53.593 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local log=/tmp/tmp.TfDn6BcQ3N/cluster1-mirror-image_44.status 2023-08-26T13:58:53.593 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_44 --format xml 2023-08-26T13:58:53.593 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-08-26T13:58:53.643 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ true 2023-08-26T13:58:53.644 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-08-26T13:58:53.645 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:-:1.1: Document is empty 2023-08-26T13:58:53.645 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr: 2023-08-26T13:58:53.645 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:^ 2023-08-26T13:58:53.646 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local primary_snapshot_id= 2023-08-26T13:58:53.646 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ test 205 = '' 2023-08-26T13:58:53.646 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-08-26T13:58:53.646 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ sleep 0.4 2023-08-26T13:58:54.047 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_44 /tmp/tmp.TfDn6BcQ3N/cluster1-mirror-image_44.status 2023-08-26T13:58:54.047 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster1 2023-08-26T13:58:54.047 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T13:58:54.048 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=image_44 2023-08-26T13:58:54.048 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local log=/tmp/tmp.TfDn6BcQ3N/cluster1-mirror-image_44.status 2023-08-26T13:58:54.048 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_44 --format xml 2023-08-26T13:58:54.048 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-08-26T13:58:54.094 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ true 2023-08-26T13:58:54.095 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-08-26T13:58:54.096 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:-:1.1: Document is empty 2023-08-26T13:58:54.096 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr: 2023-08-26T13:58:54.096 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:^ 2023-08-26T13:58:54.097 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local primary_snapshot_id= 2023-08-26T13:58:54.098 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ test 205 = '' 2023-08-26T13:58:54.098 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-08-26T13:58:54.098 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ sleep 0.8 2023-08-26T13:58:54.898 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_44 /tmp/tmp.TfDn6BcQ3N/cluster1-mirror-image_44.status 2023-08-26T13:58:54.898 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster1 2023-08-26T13:58:54.898 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T13:58:54.898 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=image_44 2023-08-26T13:58:54.899 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local log=/tmp/tmp.TfDn6BcQ3N/cluster1-mirror-image_44.status 2023-08-26T13:58:54.899 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_44 --format xml 2023-08-26T13:58:54.899 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-08-26T13:58:54.943 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-08-26T13:58:54.945 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local primary_snapshot_id=205 2023-08-26T13:58:54.945 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ test 205 = 205 2023-08-26T13:58:54.946 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ return 0 2023-08-26T13:58:54.946 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ wait_for_snap_present cluster1 mirror image_44 snap 2023-08-26T13:58:54.946 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster1 2023-08-26T13:58:54.946 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T13:58:54.946 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=image_44 2023-08-26T13:58:54.947 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local snap_name=snap 2023-08-26T13:58:54.947 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local s 2023-08-26T13:58:54.947 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2023-08-26T13:58:54.947 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ sleep 1 2023-08-26T13:58:55.946 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rbd --cluster cluster1 info mirror/image_44@snap 2023-08-26T13:58:55.990 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:rbd image 'image_44': 2023-08-26T13:58:55.991 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: size 128 MiB in 32 objects 2023-08-26T13:58:55.991 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: order 22 (4 MiB objects) 2023-08-26T13:58:55.991 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: snapshot_count: 2 2023-08-26T13:58:55.992 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: id: 12cfe4d10ff1 2023-08-26T13:58:55.992 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: block_name_prefix: rbd_data.12cfe4d10ff1 2023-08-26T13:58:55.992 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: format: 2 2023-08-26T13:58:55.992 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: features: layering, exclusive-lock, non-primary 2023-08-26T13:58:55.993 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: op_features: 2023-08-26T13:58:55.993 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: flags: 2023-08-26T13:58:55.993 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: create_timestamp: Sat Aug 26 13:50:07 2023 2023-08-26T13:58:55.993 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: access_timestamp: Sat Aug 26 13:50:07 2023 2023-08-26T13:58:55.994 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: modify_timestamp: Sat Aug 26 13:50:07 2023 2023-08-26T13:58:55.994 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: protected: False 2023-08-26T13:58:55.994 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: mirroring state: enabled 2023-08-26T13:58:55.994 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: mirroring mode: snapshot 2023-08-26T13:58:55.995 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: mirroring global id: 11ee1779-6c0f-4f0f-8432-721939d3a164 2023-08-26T13:58:55.995 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: mirroring primary: false 2023-08-26T13:58:55.996 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ return 0 2023-08-26T13:58:55.997 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ compare_image_snaps mirror image_44 snap 2023-08-26T13:58:55.997 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T13:58:55.997 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=image_44 2023-08-26T13:58:55.997 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local snap_name=snap 2023-08-26T13:58:55.997 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local ret=0 2023-08-26T13:58:55.998 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local rmt_export=/tmp/tmp.TfDn6BcQ3N/cluster2-mirror-image_44.export 2023-08-26T13:58:55.998 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local loc_export=/tmp/tmp.TfDn6BcQ3N/cluster1-mirror-image_44.export 2023-08-26T13:58:55.998 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rm -f /tmp/tmp.TfDn6BcQ3N/cluster2-mirror-image_44.export /tmp/tmp.TfDn6BcQ3N/cluster1-mirror-image_44.export 2023-08-26T13:58:55.998 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rbd --cluster cluster2 -p mirror export image_44@snap /tmp/tmp.TfDn6BcQ3N/cluster2-mirror-image_44.export 2023-08-26T13:58:56.172 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-08-26T13:58:56.514 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rbd --cluster cluster1 -p mirror export image_44@snap /tmp/tmp.TfDn6BcQ3N/cluster1-mirror-image_44.export 2023-08-26T13:58:56.514 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-08-26T13:58:56.515 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ cmp /tmp/tmp.TfDn6BcQ3N/cluster2-mirror-image_44.export /tmp/tmp.TfDn6BcQ3N/cluster1-mirror-image_44.export 2023-08-26T13:58:56.515 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rm -f /tmp/tmp.TfDn6BcQ3N/cluster2-mirror-image_44.export /tmp/tmp.TfDn6BcQ3N/cluster1-mirror-image_44.export 2023-08-26T13:58:56.515 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ return 0 2023-08-26T13:58:56.515 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-08-26T13:58:56.515 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ image=image_45 2023-08-26T13:58:56.516 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ create_snap cluster2 mirror image_45 snap 2023-08-26T13:58:56.516 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster2 2023-08-26T13:58:56.516 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T13:58:56.516 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=image_45 2023-08-26T13:58:56.517 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local snap_name=snap 2023-08-26T13:58:56.517 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rbd --cluster cluster2 -p mirror snap create image_45@snap --debug-rbd=20 --debug-journaler=20 2023-08-26T13:58:58.363 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ wait_for_image_replay_started cluster1 mirror image_45 2023-08-26T13:58:58.364 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster1 2023-08-26T13:58:58.364 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T13:58:58.364 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=image_45 2023-08-26T13:58:58.364 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ wait_for_image_replay_state cluster1 mirror image_45 started 2023-08-26T13:58:58.365 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster1 2023-08-26T13:58:58.365 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T13:58:58.365 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=image_45 2023-08-26T13:58:58.365 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local state=started 2023-08-26T13:58:58.365 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local s 2023-08-26T13:58:58.366 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-08-26T13:58:58.366 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ sleep 1 2023-08-26T13:58:59.365 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ test_image_replay_state cluster1 mirror image_45 started 2023-08-26T13:58:59.365 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster1 2023-08-26T13:58:59.366 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T13:58:59.366 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=image_45 2023-08-26T13:58:59.366 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local test_state=started 2023-08-26T13:58:59.366 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local status_result 2023-08-26T13:58:59.366 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local current_state=stopped 2023-08-26T13:58:59.370 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_45 2023-08-26T13:58:59.370 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ grep -i state 2023-08-26T13:58:59.371 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local cluster_instance=cluster1 2023-08-26T13:58:59.371 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ shift 2023-08-26T13:58:59.371 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local cluster=cluster1 2023-08-26T13:58:59.371 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local instance=cluster1 2023-08-26T13:58:59.372 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local loop_instance 2023-08-26T13:58:59.372 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-08-26T13:58:59.372 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ sleep 0 2023-08-26T13:58:59.372 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ '[' cluster1 '!=' cluster1 ']' 2023-08-26T13:58:59.372 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ seq 0 3 2023-08-26T13:58:59.373 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-08-26T13:58:59.373 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_45 2023-08-26T13:58:59.373 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local cluster=cluster1:0 2023-08-26T13:58:59.373 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ shift 2023-08-26T13:58:59.374 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local instance 2023-08-26T13:58:59.374 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ set_cluster_instance cluster1:0 cluster instance 2023-08-26T13:58:59.374 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local val=cluster1:0 2023-08-26T13:58:59.374 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local cluster_var_name=cluster 2023-08-26T13:58:59.374 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local instance_var_name=instance 2023-08-26T13:58:59.375 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ cluster=cluster1 2023-08-26T13:58:59.375 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ instance=0 2023-08-26T13:58:59.375 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ '[' 0 = cluster1:0 ']' 2023-08-26T13:58:59.375 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ eval cluster=cluster1 2023-08-26T13:58:59.375 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ cluster=cluster1 2023-08-26T13:58:59.375 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ eval instance=0 2023-08-26T13:58:59.376 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ instance=0 2023-08-26T13:58:59.376 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ daemon_asok_file cluster1:0 cluster1 2023-08-26T13:58:59.376 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ local local_cluster=cluster1:0 2023-08-26T13:58:59.376 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ local cluster=cluster1 2023-08-26T13:58:59.376 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ local instance 2023-08-26T13:58:59.377 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2023-08-26T13:58:59.377 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ local val=cluster1:0 2023-08-26T13:58:59.377 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ local cluster_var_name=local_cluster 2023-08-26T13:58:59.377 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ local instance_var_name=instance 2023-08-26T13:58:59.377 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ cluster=cluster1 2023-08-26T13:58:59.378 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ instance=0 2023-08-26T13:58:59.378 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ '[' 0 = cluster1:0 ']' 2023-08-26T13:58:59.378 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ eval local_cluster=cluster1 2023-08-26T13:58:59.378 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++++ local_cluster=cluster1 2023-08-26T13:58:59.379 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ eval instance=0 2023-08-26T13:58:59.379 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++++ instance=0 2023-08-26T13:58:59.379 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2023-08-26T13:58:59.392 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-08-26T13:58:59.393 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-08-26T13:58:59.393 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-08-26T13:58:59.393 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_45 2023-08-26T13:58:59.505 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:Invalid command: unused arguments: ['mirror/image_45'] 2023-08-26T13:58:59.506 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:rbd mirror status : get status for rbd mirror 2023-08-26T13:58:59.506 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:admin_socket: invalid command 2023-08-26T13:58:59.509 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-08-26T13:58:59.509 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_45 2023-08-26T13:58:59.509 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local cluster=cluster1:1 2023-08-26T13:58:59.509 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ shift 2023-08-26T13:58:59.510 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local instance 2023-08-26T13:58:59.510 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ set_cluster_instance cluster1:1 cluster instance 2023-08-26T13:58:59.510 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local val=cluster1:1 2023-08-26T13:58:59.510 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local cluster_var_name=cluster 2023-08-26T13:58:59.510 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local instance_var_name=instance 2023-08-26T13:58:59.511 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ cluster=cluster1 2023-08-26T13:58:59.511 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ instance=1 2023-08-26T13:58:59.511 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ '[' 1 = cluster1:1 ']' 2023-08-26T13:58:59.511 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ eval cluster=cluster1 2023-08-26T13:58:59.511 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ cluster=cluster1 2023-08-26T13:58:59.512 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ eval instance=1 2023-08-26T13:58:59.512 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ instance=1 2023-08-26T13:58:59.512 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ daemon_asok_file cluster1:1 cluster1 2023-08-26T13:58:59.512 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ local local_cluster=cluster1:1 2023-08-26T13:58:59.512 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ local cluster=cluster1 2023-08-26T13:58:59.513 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ local instance 2023-08-26T13:58:59.513 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2023-08-26T13:58:59.513 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ local val=cluster1:1 2023-08-26T13:58:59.513 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ local cluster_var_name=local_cluster 2023-08-26T13:58:59.514 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ local instance_var_name=instance 2023-08-26T13:58:59.514 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ cluster=cluster1 2023-08-26T13:58:59.514 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ instance=1 2023-08-26T13:58:59.514 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ '[' 1 = cluster1:1 ']' 2023-08-26T13:58:59.514 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ eval local_cluster=cluster1 2023-08-26T13:58:59.514 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++++ local_cluster=cluster1 2023-08-26T13:58:59.515 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ eval instance=1 2023-08-26T13:58:59.515 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++++ instance=1 2023-08-26T13:58:59.515 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2023-08-26T13:58:59.527 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-08-26T13:58:59.528 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-08-26T13:58:59.528 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-08-26T13:58:59.528 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_45 2023-08-26T13:58:59.636 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:Invalid command: unused arguments: ['mirror/image_45'] 2023-08-26T13:58:59.637 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:rbd mirror status : get status for rbd mirror 2023-08-26T13:58:59.637 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:admin_socket: invalid command 2023-08-26T13:58:59.640 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-08-26T13:58:59.641 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/image_45 2023-08-26T13:58:59.641 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local cluster=cluster1:2 2023-08-26T13:58:59.641 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ shift 2023-08-26T13:58:59.642 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local instance 2023-08-26T13:58:59.642 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ set_cluster_instance cluster1:2 cluster instance 2023-08-26T13:58:59.642 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local val=cluster1:2 2023-08-26T13:58:59.643 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local cluster_var_name=cluster 2023-08-26T13:58:59.643 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local instance_var_name=instance 2023-08-26T13:58:59.643 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ cluster=cluster1 2023-08-26T13:58:59.643 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ instance=2 2023-08-26T13:58:59.644 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ '[' 2 = cluster1:2 ']' 2023-08-26T13:58:59.644 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ eval cluster=cluster1 2023-08-26T13:58:59.644 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ cluster=cluster1 2023-08-26T13:58:59.644 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ eval instance=2 2023-08-26T13:58:59.644 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ instance=2 2023-08-26T13:58:59.644 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ daemon_asok_file cluster1:2 cluster1 2023-08-26T13:58:59.645 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ local local_cluster=cluster1:2 2023-08-26T13:58:59.645 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ local cluster=cluster1 2023-08-26T13:58:59.645 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ local instance 2023-08-26T13:58:59.645 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2023-08-26T13:58:59.645 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ local val=cluster1:2 2023-08-26T13:58:59.646 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ local cluster_var_name=local_cluster 2023-08-26T13:58:59.646 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ local instance_var_name=instance 2023-08-26T13:58:59.646 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ cluster=cluster1 2023-08-26T13:58:59.646 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ instance=2 2023-08-26T13:58:59.646 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ '[' 2 = cluster1:2 ']' 2023-08-26T13:58:59.647 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ eval local_cluster=cluster1 2023-08-26T13:58:59.647 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++++ local_cluster=cluster1 2023-08-26T13:58:59.647 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ eval instance=2 2023-08-26T13:58:59.647 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++++ instance=2 2023-08-26T13:58:59.647 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2023-08-26T13:58:59.658 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-08-26T13:58:59.659 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-08-26T13:58:59.659 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-08-26T13:58:59.659 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/image_45 2023-08-26T13:58:59.765 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:Invalid command: unused arguments: ['mirror/image_45'] 2023-08-26T13:58:59.765 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:rbd mirror status : get status for rbd mirror 2023-08-26T13:58:59.766 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:admin_socket: invalid command 2023-08-26T13:58:59.769 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-08-26T13:58:59.769 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ admin_daemon cluster1:3 rbd mirror status mirror/image_45 2023-08-26T13:58:59.769 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local cluster=cluster1:3 2023-08-26T13:58:59.769 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ shift 2023-08-26T13:58:59.769 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local instance 2023-08-26T13:58:59.770 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ set_cluster_instance cluster1:3 cluster instance 2023-08-26T13:58:59.770 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local val=cluster1:3 2023-08-26T13:58:59.770 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local cluster_var_name=cluster 2023-08-26T13:58:59.770 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local instance_var_name=instance 2023-08-26T13:58:59.770 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ cluster=cluster1 2023-08-26T13:58:59.771 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ instance=3 2023-08-26T13:58:59.771 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ '[' 3 = cluster1:3 ']' 2023-08-26T13:58:59.771 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ eval cluster=cluster1 2023-08-26T13:58:59.771 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ cluster=cluster1 2023-08-26T13:58:59.771 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ eval instance=3 2023-08-26T13:58:59.772 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ instance=3 2023-08-26T13:58:59.772 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ daemon_asok_file cluster1:3 cluster1 2023-08-26T13:58:59.772 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ local local_cluster=cluster1:3 2023-08-26T13:58:59.772 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ local cluster=cluster1 2023-08-26T13:58:59.772 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ local instance 2023-08-26T13:58:59.772 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ set_cluster_instance cluster1:3 local_cluster instance 2023-08-26T13:58:59.773 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ local val=cluster1:3 2023-08-26T13:58:59.773 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ local cluster_var_name=local_cluster 2023-08-26T13:58:59.773 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ local instance_var_name=instance 2023-08-26T13:58:59.773 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ cluster=cluster1 2023-08-26T13:58:59.773 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ instance=3 2023-08-26T13:58:59.774 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ '[' 3 = cluster1:3 ']' 2023-08-26T13:58:59.774 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ eval local_cluster=cluster1 2023-08-26T13:58:59.774 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++++ local_cluster=cluster1 2023-08-26T13:58:59.774 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ eval instance=3 2023-08-26T13:58:59.774 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++++ instance=3 2023-08-26T13:58:59.774 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2023-08-26T13:58:59.787 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2023-08-26T13:58:59.787 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2023-08-26T13:58:59.788 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2023-08-26T13:58:59.788 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror status mirror/image_45 2023-08-26T13:58:59.902 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ return 0 2023-08-26T13:58:59.903 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ status_result=' "state": "Replaying"' 2023-08-26T13:58:59.903 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ echo ' "state": "Replaying"' 2023-08-26T13:58:59.904 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ grep -i Replaying 2023-08-26T13:58:59.904 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: "state": "Replaying" 2023-08-26T13:58:59.906 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ current_state=started 2023-08-26T13:58:59.906 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ test started = started 2023-08-26T13:58:59.907 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ return 0 2023-08-26T13:58:59.907 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_45 2023-08-26T13:58:59.907 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local local_cluster=cluster1 2023-08-26T13:58:59.907 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster2 2023-08-26T13:58:59.907 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T13:58:59.908 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=image_45 2023-08-26T13:58:59.908 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ '[' snapshot = journal ']' 2023-08-26T13:58:59.908 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ '[' snapshot = snapshot ']' 2023-08-26T13:58:59.908 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_45 2023-08-26T13:58:59.909 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local local_cluster=cluster1 2023-08-26T13:58:59.909 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster2 2023-08-26T13:58:59.909 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T13:58:59.909 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=image_45 2023-08-26T13:58:59.909 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ mkfname cluster2-mirror-image_45.status 2023-08-26T13:58:59.910 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ echo cluster2-mirror-image_45.status 2023-08-26T13:58:59.910 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ sed -e 's|[/ ]|_|g' 2023-08-26T13:58:59.910 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local status_log=/tmp/tmp.TfDn6BcQ3N/cluster2-mirror-image_45.status 2023-08-26T13:58:59.910 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ mkfname cluster1-mirror-image_45.status 2023-08-26T13:58:59.910 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ echo cluster1-mirror-image_45.status 2023-08-26T13:58:59.911 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ sed -e 's|[/ ]|_|g' 2023-08-26T13:58:59.911 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local local_status_log=/tmp/tmp.TfDn6BcQ3N/cluster1-mirror-image_45.status 2023-08-26T13:58:59.912 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ mirror_image_snapshot cluster2 mirror image_45 2023-08-26T13:58:59.912 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster2 2023-08-26T13:58:59.912 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T13:58:59.912 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=image_45 2023-08-26T13:58:59.912 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_45 2023-08-26T13:59:00.366 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:Snapshot ID: 207 2023-08-26T13:59:00.375 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_45 /tmp/tmp.TfDn6BcQ3N/cluster2-mirror-image_45.status 2023-08-26T13:59:00.375 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster2 2023-08-26T13:59:00.375 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T13:59:00.375 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=image_45 2023-08-26T13:59:00.376 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local log=/tmp/tmp.TfDn6BcQ3N/cluster2-mirror-image_45.status 2023-08-26T13:59:00.376 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_45 --format xml 2023-08-26T13:59:00.376 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-08-26T13:59:00.417 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ xmlstarlet sel -t -v //snapshot/id 2023-08-26T13:59:00.419 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local snapshot_id=207 2023-08-26T13:59:00.419 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ true 2023-08-26T13:59:00.419 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-08-26T13:59:00.419 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ sleep 0.2 2023-08-26T13:59:00.620 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_45 /tmp/tmp.TfDn6BcQ3N/cluster1-mirror-image_45.status 2023-08-26T13:59:00.621 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster1 2023-08-26T13:59:00.621 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T13:59:00.621 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=image_45 2023-08-26T13:59:00.621 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local log=/tmp/tmp.TfDn6BcQ3N/cluster1-mirror-image_45.status 2023-08-26T13:59:00.622 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_45 --format xml 2023-08-26T13:59:00.622 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-08-26T13:59:00.672 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-08-26T13:59:00.674 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local primary_snapshot_id=112 2023-08-26T13:59:00.674 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ test 207 = 112 2023-08-26T13:59:00.674 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-08-26T13:59:00.674 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ sleep 0.4 2023-08-26T13:59:01.075 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_45 /tmp/tmp.TfDn6BcQ3N/cluster1-mirror-image_45.status 2023-08-26T13:59:01.076 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster1 2023-08-26T13:59:01.076 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T13:59:01.076 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=image_45 2023-08-26T13:59:01.077 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local log=/tmp/tmp.TfDn6BcQ3N/cluster1-mirror-image_45.status 2023-08-26T13:59:01.077 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_45 --format xml 2023-08-26T13:59:01.077 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-08-26T13:59:01.124 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-08-26T13:59:01.126 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local primary_snapshot_id=112 2023-08-26T13:59:01.126 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ test 207 = 112 2023-08-26T13:59:01.126 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-08-26T13:59:01.126 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ sleep 0.8 2023-08-26T13:59:01.927 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_45 /tmp/tmp.TfDn6BcQ3N/cluster1-mirror-image_45.status 2023-08-26T13:59:01.928 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster1 2023-08-26T13:59:01.928 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T13:59:01.928 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=image_45 2023-08-26T13:59:01.928 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local log=/tmp/tmp.TfDn6BcQ3N/cluster1-mirror-image_45.status 2023-08-26T13:59:01.929 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_45 --format xml 2023-08-26T13:59:01.929 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-08-26T13:59:01.974 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ true 2023-08-26T13:59:01.974 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-08-26T13:59:01.976 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:-:1.1: Document is empty 2023-08-26T13:59:01.976 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr: 2023-08-26T13:59:01.976 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:^ 2023-08-26T13:59:01.976 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local primary_snapshot_id= 2023-08-26T13:59:01.976 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ test 207 = '' 2023-08-26T13:59:01.977 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-08-26T13:59:01.977 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ sleep 1.6 2023-08-26T13:59:03.577 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_45 /tmp/tmp.TfDn6BcQ3N/cluster1-mirror-image_45.status 2023-08-26T13:59:03.578 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster1 2023-08-26T13:59:03.578 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T13:59:03.578 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=image_45 2023-08-26T13:59:03.578 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local log=/tmp/tmp.TfDn6BcQ3N/cluster1-mirror-image_45.status 2023-08-26T13:59:03.579 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_45 --format xml 2023-08-26T13:59:03.579 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-08-26T13:59:03.624 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-08-26T13:59:03.626 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local primary_snapshot_id=207 2023-08-26T13:59:03.626 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ test 207 = 207 2023-08-26T13:59:03.627 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ return 0 2023-08-26T13:59:03.627 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ wait_for_snap_present cluster1 mirror image_45 snap 2023-08-26T13:59:03.627 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster1 2023-08-26T13:59:03.627 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T13:59:03.627 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=image_45 2023-08-26T13:59:03.628 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local snap_name=snap 2023-08-26T13:59:03.628 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local s 2023-08-26T13:59:03.628 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2023-08-26T13:59:03.628 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ sleep 1 2023-08-26T13:59:04.627 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rbd --cluster cluster1 info mirror/image_45@snap 2023-08-26T13:59:04.672 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:rbd image 'image_45': 2023-08-26T13:59:04.672 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: size 128 MiB in 32 objects 2023-08-26T13:59:04.673 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: order 22 (4 MiB objects) 2023-08-26T13:59:04.673 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: snapshot_count: 2 2023-08-26T13:59:04.673 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: id: 10b760c53e79 2023-08-26T13:59:04.673 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: block_name_prefix: rbd_data.10b760c53e79 2023-08-26T13:59:04.674 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: format: 2 2023-08-26T13:59:04.674 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: features: layering, exclusive-lock, non-primary 2023-08-26T13:59:04.674 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: op_features: 2023-08-26T13:59:04.674 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: flags: 2023-08-26T13:59:04.674 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: create_timestamp: Sat Aug 26 13:50:18 2023 2023-08-26T13:59:04.675 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: access_timestamp: Sat Aug 26 13:50:18 2023 2023-08-26T13:59:04.675 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: modify_timestamp: Sat Aug 26 13:50:18 2023 2023-08-26T13:59:04.675 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: protected: False 2023-08-26T13:59:04.675 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: mirroring state: enabled 2023-08-26T13:59:04.676 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: mirroring mode: snapshot 2023-08-26T13:59:04.676 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: mirroring global id: 1822564f-3eee-4ba8-b76e-97584ef869b2 2023-08-26T13:59:04.676 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: mirroring primary: false 2023-08-26T13:59:04.677 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ return 0 2023-08-26T13:59:04.678 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ compare_image_snaps mirror image_45 snap 2023-08-26T13:59:04.678 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T13:59:04.678 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=image_45 2023-08-26T13:59:04.678 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local snap_name=snap 2023-08-26T13:59:04.679 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local ret=0 2023-08-26T13:59:04.679 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local rmt_export=/tmp/tmp.TfDn6BcQ3N/cluster2-mirror-image_45.export 2023-08-26T13:59:04.679 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local loc_export=/tmp/tmp.TfDn6BcQ3N/cluster1-mirror-image_45.export 2023-08-26T13:59:04.679 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rm -f /tmp/tmp.TfDn6BcQ3N/cluster2-mirror-image_45.export /tmp/tmp.TfDn6BcQ3N/cluster1-mirror-image_45.export 2023-08-26T13:59:04.679 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rbd --cluster cluster2 -p mirror export image_45@snap /tmp/tmp.TfDn6BcQ3N/cluster2-mirror-image_45.export 2023-08-26T13:59:04.858 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-08-26T13:59:04.863 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rbd --cluster cluster1 -p mirror export image_45@snap /tmp/tmp.TfDn6BcQ3N/cluster1-mirror-image_45.export 2023-08-26T13:59:05.098 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-08-26T13:59:05.103 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ cmp /tmp/tmp.TfDn6BcQ3N/cluster2-mirror-image_45.export /tmp/tmp.TfDn6BcQ3N/cluster1-mirror-image_45.export 2023-08-26T13:59:05.159 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rm -f /tmp/tmp.TfDn6BcQ3N/cluster2-mirror-image_45.export /tmp/tmp.TfDn6BcQ3N/cluster1-mirror-image_45.export 2023-08-26T13:59:05.198 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ return 0 2023-08-26T13:59:05.198 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-08-26T13:59:05.198 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ image=image_46 2023-08-26T13:59:05.199 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ create_snap cluster2 mirror image_46 snap 2023-08-26T13:59:05.199 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster2 2023-08-26T13:59:05.199 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T13:59:05.199 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=image_46 2023-08-26T13:59:05.200 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local snap_name=snap 2023-08-26T13:59:05.200 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rbd --cluster cluster2 -p mirror snap create image_46@snap --debug-rbd=20 --debug-journaler=20 2023-08-26T13:59:07.227 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ wait_for_image_replay_started cluster1 mirror image_46 2023-08-26T13:59:07.228 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster1 2023-08-26T13:59:07.228 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T13:59:07.228 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=image_46 2023-08-26T13:59:07.228 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ wait_for_image_replay_state cluster1 mirror image_46 started 2023-08-26T13:59:07.228 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster1 2023-08-26T13:59:07.229 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T13:59:07.229 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=image_46 2023-08-26T13:59:07.229 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local state=started 2023-08-26T13:59:07.229 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local s 2023-08-26T13:59:07.229 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-08-26T13:59:07.230 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ sleep 1 2023-08-26T13:59:08.228 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ test_image_replay_state cluster1 mirror image_46 started 2023-08-26T13:59:08.229 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster1 2023-08-26T13:59:08.229 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T13:59:08.229 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=image_46 2023-08-26T13:59:08.229 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local test_state=started 2023-08-26T13:59:08.229 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local status_result 2023-08-26T13:59:08.230 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local current_state=stopped 2023-08-26T13:59:08.232 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_46 2023-08-26T13:59:08.232 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local cluster_instance=cluster1 2023-08-26T13:59:08.232 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ shift 2023-08-26T13:59:08.232 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ grep -i state 2023-08-26T13:59:08.232 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local cluster=cluster1 2023-08-26T13:59:08.233 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local instance=cluster1 2023-08-26T13:59:08.233 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local loop_instance 2023-08-26T13:59:08.233 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-08-26T13:59:08.233 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ sleep 0 2023-08-26T13:59:08.233 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ '[' cluster1 '!=' cluster1 ']' 2023-08-26T13:59:08.234 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ seq 0 3 2023-08-26T13:59:08.236 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-08-26T13:59:08.236 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_46 2023-08-26T13:59:08.237 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local cluster=cluster1:0 2023-08-26T13:59:08.237 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ shift 2023-08-26T13:59:08.237 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local instance 2023-08-26T13:59:08.237 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ set_cluster_instance cluster1:0 cluster instance 2023-08-26T13:59:08.237 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local val=cluster1:0 2023-08-26T13:59:08.238 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local cluster_var_name=cluster 2023-08-26T13:59:08.238 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local instance_var_name=instance 2023-08-26T13:59:08.238 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ cluster=cluster1 2023-08-26T13:59:08.238 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ instance=0 2023-08-26T13:59:08.238 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ '[' 0 = cluster1:0 ']' 2023-08-26T13:59:08.239 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ eval cluster=cluster1 2023-08-26T13:59:08.239 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ cluster=cluster1 2023-08-26T13:59:08.239 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ eval instance=0 2023-08-26T13:59:08.239 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ instance=0 2023-08-26T13:59:08.239 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ daemon_asok_file cluster1:0 cluster1 2023-08-26T13:59:08.240 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ local local_cluster=cluster1:0 2023-08-26T13:59:08.240 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ local cluster=cluster1 2023-08-26T13:59:08.240 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ local instance 2023-08-26T13:59:08.240 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2023-08-26T13:59:08.240 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ local val=cluster1:0 2023-08-26T13:59:08.240 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ local cluster_var_name=local_cluster 2023-08-26T13:59:08.241 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ local instance_var_name=instance 2023-08-26T13:59:08.241 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ cluster=cluster1 2023-08-26T13:59:08.241 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ instance=0 2023-08-26T13:59:08.241 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ '[' 0 = cluster1:0 ']' 2023-08-26T13:59:08.241 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ eval local_cluster=cluster1 2023-08-26T13:59:08.242 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++++ local_cluster=cluster1 2023-08-26T13:59:08.242 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ eval instance=0 2023-08-26T13:59:08.242 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++++ instance=0 2023-08-26T13:59:08.242 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2023-08-26T13:59:08.255 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-08-26T13:59:08.256 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-08-26T13:59:08.256 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-08-26T13:59:08.256 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_46 2023-08-26T13:59:08.372 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:Invalid command: unused arguments: ['mirror/image_46'] 2023-08-26T13:59:08.373 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:rbd mirror status : get status for rbd mirror 2023-08-26T13:59:08.373 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:admin_socket: invalid command 2023-08-26T13:59:08.376 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-08-26T13:59:08.376 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_46 2023-08-26T13:59:08.376 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local cluster=cluster1:1 2023-08-26T13:59:08.376 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ shift 2023-08-26T13:59:08.376 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local instance 2023-08-26T13:59:08.377 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ set_cluster_instance cluster1:1 cluster instance 2023-08-26T13:59:08.377 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local val=cluster1:1 2023-08-26T13:59:08.377 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local cluster_var_name=cluster 2023-08-26T13:59:08.377 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local instance_var_name=instance 2023-08-26T13:59:08.378 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ cluster=cluster1 2023-08-26T13:59:08.378 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ instance=1 2023-08-26T13:59:08.378 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ '[' 1 = cluster1:1 ']' 2023-08-26T13:59:08.378 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ eval cluster=cluster1 2023-08-26T13:59:08.378 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ cluster=cluster1 2023-08-26T13:59:08.379 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ eval instance=1 2023-08-26T13:59:08.379 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ instance=1 2023-08-26T13:59:08.379 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ daemon_asok_file cluster1:1 cluster1 2023-08-26T13:59:08.379 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ local local_cluster=cluster1:1 2023-08-26T13:59:08.379 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ local cluster=cluster1 2023-08-26T13:59:08.380 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ local instance 2023-08-26T13:59:08.380 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2023-08-26T13:59:08.380 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ local val=cluster1:1 2023-08-26T13:59:08.380 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ local cluster_var_name=local_cluster 2023-08-26T13:59:08.380 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ local instance_var_name=instance 2023-08-26T13:59:08.380 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ cluster=cluster1 2023-08-26T13:59:08.381 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ instance=1 2023-08-26T13:59:08.381 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ '[' 1 = cluster1:1 ']' 2023-08-26T13:59:08.381 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ eval local_cluster=cluster1 2023-08-26T13:59:08.381 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++++ local_cluster=cluster1 2023-08-26T13:59:08.381 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ eval instance=1 2023-08-26T13:59:08.382 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++++ instance=1 2023-08-26T13:59:08.382 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2023-08-26T13:59:08.395 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-08-26T13:59:08.395 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-08-26T13:59:08.395 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-08-26T13:59:08.395 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_46 2023-08-26T13:59:08.503 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:Invalid command: unused arguments: ['mirror/image_46'] 2023-08-26T13:59:08.504 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:rbd mirror status : get status for rbd mirror 2023-08-26T13:59:08.504 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:admin_socket: invalid command 2023-08-26T13:59:08.507 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-08-26T13:59:08.507 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/image_46 2023-08-26T13:59:08.507 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local cluster=cluster1:2 2023-08-26T13:59:08.507 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ shift 2023-08-26T13:59:08.507 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local instance 2023-08-26T13:59:08.508 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ set_cluster_instance cluster1:2 cluster instance 2023-08-26T13:59:08.508 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local val=cluster1:2 2023-08-26T13:59:08.508 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local cluster_var_name=cluster 2023-08-26T13:59:08.508 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local instance_var_name=instance 2023-08-26T13:59:08.508 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ cluster=cluster1 2023-08-26T13:59:08.509 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ instance=2 2023-08-26T13:59:08.509 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ '[' 2 = cluster1:2 ']' 2023-08-26T13:59:08.509 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ eval cluster=cluster1 2023-08-26T13:59:08.509 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ cluster=cluster1 2023-08-26T13:59:08.509 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ eval instance=2 2023-08-26T13:59:08.510 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ instance=2 2023-08-26T13:59:08.510 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ daemon_asok_file cluster1:2 cluster1 2023-08-26T13:59:08.510 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ local local_cluster=cluster1:2 2023-08-26T13:59:08.510 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ local cluster=cluster1 2023-08-26T13:59:08.510 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ local instance 2023-08-26T13:59:08.511 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2023-08-26T13:59:08.511 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ local val=cluster1:2 2023-08-26T13:59:08.511 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ local cluster_var_name=local_cluster 2023-08-26T13:59:08.511 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ local instance_var_name=instance 2023-08-26T13:59:08.511 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ cluster=cluster1 2023-08-26T13:59:08.512 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ instance=2 2023-08-26T13:59:08.512 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ '[' 2 = cluster1:2 ']' 2023-08-26T13:59:08.512 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ eval local_cluster=cluster1 2023-08-26T13:59:08.512 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++++ local_cluster=cluster1 2023-08-26T13:59:08.512 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ eval instance=2 2023-08-26T13:59:08.513 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++++ instance=2 2023-08-26T13:59:08.513 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2023-08-26T13:59:08.525 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-08-26T13:59:08.525 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-08-26T13:59:08.525 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-08-26T13:59:08.526 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/image_46 2023-08-26T13:59:08.630 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:Invalid command: unused arguments: ['mirror/image_46'] 2023-08-26T13:59:08.631 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:rbd mirror status : get status for rbd mirror 2023-08-26T13:59:08.631 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:admin_socket: invalid command 2023-08-26T13:59:08.634 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-08-26T13:59:08.634 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ admin_daemon cluster1:3 rbd mirror status mirror/image_46 2023-08-26T13:59:08.634 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local cluster=cluster1:3 2023-08-26T13:59:08.634 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ shift 2023-08-26T13:59:08.634 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local instance 2023-08-26T13:59:08.635 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ set_cluster_instance cluster1:3 cluster instance 2023-08-26T13:59:08.635 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local val=cluster1:3 2023-08-26T13:59:08.635 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local cluster_var_name=cluster 2023-08-26T13:59:08.635 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local instance_var_name=instance 2023-08-26T13:59:08.635 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ cluster=cluster1 2023-08-26T13:59:08.636 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ instance=3 2023-08-26T13:59:08.636 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ '[' 3 = cluster1:3 ']' 2023-08-26T13:59:08.636 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ eval cluster=cluster1 2023-08-26T13:59:08.636 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ cluster=cluster1 2023-08-26T13:59:08.636 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ eval instance=3 2023-08-26T13:59:08.637 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ instance=3 2023-08-26T13:59:08.637 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ daemon_asok_file cluster1:3 cluster1 2023-08-26T13:59:08.637 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ local local_cluster=cluster1:3 2023-08-26T13:59:08.637 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ local cluster=cluster1 2023-08-26T13:59:08.637 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ local instance 2023-08-26T13:59:08.638 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ set_cluster_instance cluster1:3 local_cluster instance 2023-08-26T13:59:08.638 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ local val=cluster1:3 2023-08-26T13:59:08.638 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ local cluster_var_name=local_cluster 2023-08-26T13:59:08.638 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ local instance_var_name=instance 2023-08-26T13:59:08.638 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ cluster=cluster1 2023-08-26T13:59:08.639 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ instance=3 2023-08-26T13:59:08.639 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ '[' 3 = cluster1:3 ']' 2023-08-26T13:59:08.639 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ eval local_cluster=cluster1 2023-08-26T13:59:08.639 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++++ local_cluster=cluster1 2023-08-26T13:59:08.639 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ eval instance=3 2023-08-26T13:59:08.640 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++++ instance=3 2023-08-26T13:59:08.640 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2023-08-26T13:59:08.652 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2023-08-26T13:59:08.653 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2023-08-26T13:59:08.653 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2023-08-26T13:59:08.653 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror status mirror/image_46 2023-08-26T13:59:08.767 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ return 0 2023-08-26T13:59:08.768 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ status_result=' "state": "Replaying"' 2023-08-26T13:59:08.768 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ echo ' "state": "Replaying"' 2023-08-26T13:59:08.769 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ grep -i Replaying 2023-08-26T13:59:08.769 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: "state": "Replaying" 2023-08-26T13:59:08.771 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ current_state=started 2023-08-26T13:59:08.771 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ test started = started 2023-08-26T13:59:08.772 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ return 0 2023-08-26T13:59:08.772 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_46 2023-08-26T13:59:08.772 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local local_cluster=cluster1 2023-08-26T13:59:08.773 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster2 2023-08-26T13:59:08.773 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T13:59:08.773 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=image_46 2023-08-26T13:59:08.773 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ '[' snapshot = journal ']' 2023-08-26T13:59:08.773 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ '[' snapshot = snapshot ']' 2023-08-26T13:59:08.774 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_46 2023-08-26T13:59:08.774 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local local_cluster=cluster1 2023-08-26T13:59:08.774 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster2 2023-08-26T13:59:08.775 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T13:59:08.776 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=image_46 2023-08-26T13:59:08.776 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ mkfname cluster2-mirror-image_46.status 2023-08-26T13:59:08.776 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ echo cluster2-mirror-image_46.status 2023-08-26T13:59:08.777 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ sed -e 's|[/ ]|_|g' 2023-08-26T13:59:08.777 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local status_log=/tmp/tmp.TfDn6BcQ3N/cluster2-mirror-image_46.status 2023-08-26T13:59:08.777 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ mkfname cluster1-mirror-image_46.status 2023-08-26T13:59:08.777 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ echo cluster1-mirror-image_46.status 2023-08-26T13:59:08.778 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ sed -e 's|[/ ]|_|g' 2023-08-26T13:59:08.778 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local local_status_log=/tmp/tmp.TfDn6BcQ3N/cluster1-mirror-image_46.status 2023-08-26T13:59:08.779 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ mirror_image_snapshot cluster2 mirror image_46 2023-08-26T13:59:08.779 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster2 2023-08-26T13:59:08.779 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T13:59:08.779 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=image_46 2023-08-26T13:59:08.780 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_46 2023-08-26T13:59:09.228 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:Snapshot ID: 209 2023-08-26T13:59:09.237 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_46 /tmp/tmp.TfDn6BcQ3N/cluster2-mirror-image_46.status 2023-08-26T13:59:09.237 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster2 2023-08-26T13:59:09.237 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T13:59:09.237 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=image_46 2023-08-26T13:59:09.237 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local log=/tmp/tmp.TfDn6BcQ3N/cluster2-mirror-image_46.status 2023-08-26T13:59:09.238 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_46 --format xml 2023-08-26T13:59:09.238 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-08-26T13:59:09.279 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ xmlstarlet sel -t -v //snapshot/id 2023-08-26T13:59:09.281 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local snapshot_id=209 2023-08-26T13:59:09.281 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ true 2023-08-26T13:59:09.281 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-08-26T13:59:09.281 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ sleep 0.2 2023-08-26T13:59:09.482 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_46 /tmp/tmp.TfDn6BcQ3N/cluster1-mirror-image_46.status 2023-08-26T13:59:09.482 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster1 2023-08-26T13:59:09.483 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T13:59:09.483 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=image_46 2023-08-26T13:59:09.483 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local log=/tmp/tmp.TfDn6BcQ3N/cluster1-mirror-image_46.status 2023-08-26T13:59:09.483 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_46 --format xml 2023-08-26T13:59:09.483 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-08-26T13:59:09.529 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-08-26T13:59:09.531 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local primary_snapshot_id=113 2023-08-26T13:59:09.531 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ test 209 = 113 2023-08-26T13:59:09.531 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-08-26T13:59:09.531 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ sleep 0.4 2023-08-26T13:59:09.932 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_46 /tmp/tmp.TfDn6BcQ3N/cluster1-mirror-image_46.status 2023-08-26T13:59:09.933 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster1 2023-08-26T13:59:09.933 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T13:59:09.933 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=image_46 2023-08-26T13:59:09.933 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local log=/tmp/tmp.TfDn6BcQ3N/cluster1-mirror-image_46.status 2023-08-26T13:59:09.934 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_46 --format xml 2023-08-26T13:59:09.934 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-08-26T13:59:09.977 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ true 2023-08-26T13:59:09.978 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-08-26T13:59:09.978 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:-:1.1: Document is empty 2023-08-26T13:59:09.979 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr: 2023-08-26T13:59:09.979 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:^ 2023-08-26T13:59:09.980 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local primary_snapshot_id= 2023-08-26T13:59:09.980 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ test 209 = '' 2023-08-26T13:59:09.980 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-08-26T13:59:09.980 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ sleep 0.8 2023-08-26T13:59:10.780 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_46 /tmp/tmp.TfDn6BcQ3N/cluster1-mirror-image_46.status 2023-08-26T13:59:10.781 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster1 2023-08-26T13:59:10.781 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T13:59:10.781 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=image_46 2023-08-26T13:59:10.781 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local log=/tmp/tmp.TfDn6BcQ3N/cluster1-mirror-image_46.status 2023-08-26T13:59:10.782 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_46 --format xml 2023-08-26T13:59:10.782 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-08-26T13:59:10.824 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-08-26T13:59:10.826 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local primary_snapshot_id=209 2023-08-26T13:59:10.827 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ test 209 = 209 2023-08-26T13:59:10.827 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ return 0 2023-08-26T13:59:10.827 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ wait_for_snap_present cluster1 mirror image_46 snap 2023-08-26T13:59:10.827 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster1 2023-08-26T13:59:10.828 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T13:59:10.828 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=image_46 2023-08-26T13:59:10.828 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local snap_name=snap 2023-08-26T13:59:10.828 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local s 2023-08-26T13:59:10.828 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2023-08-26T13:59:10.829 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ sleep 1 2023-08-26T13:59:11.827 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rbd --cluster cluster1 info mirror/image_46@snap 2023-08-26T13:59:11.872 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:rbd image 'image_46': 2023-08-26T13:59:11.872 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: size 128 MiB in 32 objects 2023-08-26T13:59:11.872 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: order 22 (4 MiB objects) 2023-08-26T13:59:11.873 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: snapshot_count: 2 2023-08-26T13:59:11.873 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: id: 122bcb344b6f 2023-08-26T13:59:11.873 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: block_name_prefix: rbd_data.122bcb344b6f 2023-08-26T13:59:11.873 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: format: 2 2023-08-26T13:59:11.873 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: features: layering, exclusive-lock, non-primary 2023-08-26T13:59:11.873 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: op_features: 2023-08-26T13:59:11.874 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: flags: 2023-08-26T13:59:11.874 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: create_timestamp: Sat Aug 26 13:50:25 2023 2023-08-26T13:59:11.874 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: access_timestamp: Sat Aug 26 13:50:25 2023 2023-08-26T13:59:11.874 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: modify_timestamp: Sat Aug 26 13:50:25 2023 2023-08-26T13:59:11.874 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: protected: False 2023-08-26T13:59:11.875 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: mirroring state: enabled 2023-08-26T13:59:11.875 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: mirroring mode: snapshot 2023-08-26T13:59:11.875 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: mirroring global id: 994498aa-fae7-419f-8291-f66b876aef55 2023-08-26T13:59:11.875 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: mirroring primary: false 2023-08-26T13:59:11.876 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ return 0 2023-08-26T13:59:11.877 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ compare_image_snaps mirror image_46 snap 2023-08-26T13:59:11.877 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T13:59:11.877 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=image_46 2023-08-26T13:59:11.877 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local snap_name=snap 2023-08-26T13:59:11.877 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local ret=0 2023-08-26T13:59:11.878 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local rmt_export=/tmp/tmp.TfDn6BcQ3N/cluster2-mirror-image_46.export 2023-08-26T13:59:11.878 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local loc_export=/tmp/tmp.TfDn6BcQ3N/cluster1-mirror-image_46.export 2023-08-26T13:59:11.878 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rm -f /tmp/tmp.TfDn6BcQ3N/cluster2-mirror-image_46.export /tmp/tmp.TfDn6BcQ3N/cluster1-mirror-image_46.export 2023-08-26T13:59:11.878 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rbd --cluster cluster2 -p mirror export image_46@snap /tmp/tmp.TfDn6BcQ3N/cluster2-mirror-image_46.export 2023-08-26T13:59:12.053 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-08-26T13:59:12.060 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rbd --cluster cluster1 -p mirror export image_46@snap /tmp/tmp.TfDn6BcQ3N/cluster1-mirror-image_46.export 2023-08-26T13:59:12.303 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-08-26T13:59:12.311 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ cmp /tmp/tmp.TfDn6BcQ3N/cluster2-mirror-image_46.export /tmp/tmp.TfDn6BcQ3N/cluster1-mirror-image_46.export 2023-08-26T13:59:12.365 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rm -f /tmp/tmp.TfDn6BcQ3N/cluster2-mirror-image_46.export /tmp/tmp.TfDn6BcQ3N/cluster1-mirror-image_46.export 2023-08-26T13:59:12.402 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ return 0 2023-08-26T13:59:12.403 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-08-26T13:59:12.403 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ image=image_47 2023-08-26T13:59:12.403 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ create_snap cluster2 mirror image_47 snap 2023-08-26T13:59:12.403 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster2 2023-08-26T13:59:12.404 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T13:59:12.404 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=image_47 2023-08-26T13:59:12.404 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local snap_name=snap 2023-08-26T13:59:12.404 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rbd --cluster cluster2 -p mirror snap create image_47@snap --debug-rbd=20 --debug-journaler=20 2023-08-26T13:59:14.310 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ wait_for_image_replay_started cluster1 mirror image_47 2023-08-26T13:59:14.310 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster1 2023-08-26T13:59:14.310 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T13:59:14.311 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=image_47 2023-08-26T13:59:14.311 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ wait_for_image_replay_state cluster1 mirror image_47 started 2023-08-26T13:59:14.311 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster1 2023-08-26T13:59:14.311 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T13:59:14.311 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=image_47 2023-08-26T13:59:14.312 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local state=started 2023-08-26T13:59:14.312 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local s 2023-08-26T13:59:14.312 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-08-26T13:59:14.312 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ sleep 1 2023-08-26T13:59:15.312 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ test_image_replay_state cluster1 mirror image_47 started 2023-08-26T13:59:15.312 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster1 2023-08-26T13:59:15.313 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T13:59:15.313 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=image_47 2023-08-26T13:59:15.313 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local test_state=started 2023-08-26T13:59:15.313 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local status_result 2023-08-26T13:59:15.314 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local current_state=stopped 2023-08-26T13:59:15.314 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_47 2023-08-26T13:59:15.314 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ grep -i state 2023-08-26T13:59:15.314 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local cluster_instance=cluster1 2023-08-26T13:59:15.315 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ shift 2023-08-26T13:59:15.315 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local cluster=cluster1 2023-08-26T13:59:15.315 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local instance=cluster1 2023-08-26T13:59:15.315 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local loop_instance 2023-08-26T13:59:15.315 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-08-26T13:59:15.316 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ sleep 0 2023-08-26T13:59:15.319 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ '[' cluster1 '!=' cluster1 ']' 2023-08-26T13:59:15.319 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ seq 0 3 2023-08-26T13:59:15.319 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-08-26T13:59:15.320 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_47 2023-08-26T13:59:15.320 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local cluster=cluster1:0 2023-08-26T13:59:15.320 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ shift 2023-08-26T13:59:15.320 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local instance 2023-08-26T13:59:15.320 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ set_cluster_instance cluster1:0 cluster instance 2023-08-26T13:59:15.321 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local val=cluster1:0 2023-08-26T13:59:15.321 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local cluster_var_name=cluster 2023-08-26T13:59:15.321 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local instance_var_name=instance 2023-08-26T13:59:15.321 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ cluster=cluster1 2023-08-26T13:59:15.321 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ instance=0 2023-08-26T13:59:15.322 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ '[' 0 = cluster1:0 ']' 2023-08-26T13:59:15.322 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ eval cluster=cluster1 2023-08-26T13:59:15.322 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ cluster=cluster1 2023-08-26T13:59:15.322 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ eval instance=0 2023-08-26T13:59:15.322 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ instance=0 2023-08-26T13:59:15.323 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ daemon_asok_file cluster1:0 cluster1 2023-08-26T13:59:15.323 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ local local_cluster=cluster1:0 2023-08-26T13:59:15.323 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ local cluster=cluster1 2023-08-26T13:59:15.323 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ local instance 2023-08-26T13:59:15.324 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2023-08-26T13:59:15.324 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ local val=cluster1:0 2023-08-26T13:59:15.324 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ local cluster_var_name=local_cluster 2023-08-26T13:59:15.324 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ local instance_var_name=instance 2023-08-26T13:59:15.324 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ cluster=cluster1 2023-08-26T13:59:15.325 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ instance=0 2023-08-26T13:59:15.325 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ '[' 0 = cluster1:0 ']' 2023-08-26T13:59:15.325 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ eval local_cluster=cluster1 2023-08-26T13:59:15.325 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++++ local_cluster=cluster1 2023-08-26T13:59:15.325 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ eval instance=0 2023-08-26T13:59:15.326 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++++ instance=0 2023-08-26T13:59:15.326 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2023-08-26T13:59:15.338 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-08-26T13:59:15.338 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-08-26T13:59:15.339 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-08-26T13:59:15.339 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_47 2023-08-26T13:59:15.462 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ return 0 2023-08-26T13:59:15.463 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ status_result=' "state": "Replaying"' 2023-08-26T13:59:15.464 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ echo ' "state": "Replaying"' 2023-08-26T13:59:15.464 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ grep -i Replaying 2023-08-26T13:59:15.464 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: "state": "Replaying" 2023-08-26T13:59:15.466 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ current_state=started 2023-08-26T13:59:15.467 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ test started = started 2023-08-26T13:59:15.467 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ return 0 2023-08-26T13:59:15.467 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_47 2023-08-26T13:59:15.467 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local local_cluster=cluster1 2023-08-26T13:59:15.468 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster2 2023-08-26T13:59:15.468 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T13:59:15.468 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=image_47 2023-08-26T13:59:15.468 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ '[' snapshot = journal ']' 2023-08-26T13:59:15.468 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ '[' snapshot = snapshot ']' 2023-08-26T13:59:15.469 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_47 2023-08-26T13:59:15.469 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local local_cluster=cluster1 2023-08-26T13:59:15.469 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster2 2023-08-26T13:59:15.469 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T13:59:15.469 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=image_47 2023-08-26T13:59:15.469 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ mkfname cluster2-mirror-image_47.status 2023-08-26T13:59:15.470 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ echo cluster2-mirror-image_47.status 2023-08-26T13:59:15.470 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ sed -e 's|[/ ]|_|g' 2023-08-26T13:59:15.470 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local status_log=/tmp/tmp.TfDn6BcQ3N/cluster2-mirror-image_47.status 2023-08-26T13:59:15.470 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ mkfname cluster1-mirror-image_47.status 2023-08-26T13:59:15.471 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ echo cluster1-mirror-image_47.status 2023-08-26T13:59:15.471 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ sed -e 's|[/ ]|_|g' 2023-08-26T13:59:15.472 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local local_status_log=/tmp/tmp.TfDn6BcQ3N/cluster1-mirror-image_47.status 2023-08-26T13:59:15.472 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ mirror_image_snapshot cluster2 mirror image_47 2023-08-26T13:59:15.472 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster2 2023-08-26T13:59:15.472 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T13:59:15.472 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=image_47 2023-08-26T13:59:15.473 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_47 2023-08-26T13:59:16.317 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:Snapshot ID: 211 2023-08-26T13:59:16.327 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_47 /tmp/tmp.TfDn6BcQ3N/cluster2-mirror-image_47.status 2023-08-26T13:59:16.328 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster2 2023-08-26T13:59:16.328 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T13:59:16.328 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=image_47 2023-08-26T13:59:16.328 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local log=/tmp/tmp.TfDn6BcQ3N/cluster2-mirror-image_47.status 2023-08-26T13:59:16.328 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_47 --format xml 2023-08-26T13:59:16.329 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-08-26T13:59:16.372 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ xmlstarlet sel -t -v //snapshot/id 2023-08-26T13:59:16.374 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local snapshot_id=211 2023-08-26T13:59:16.374 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ true 2023-08-26T13:59:16.374 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-08-26T13:59:16.374 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ sleep 0.2 2023-08-26T13:59:16.575 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_47 /tmp/tmp.TfDn6BcQ3N/cluster1-mirror-image_47.status 2023-08-26T13:59:16.576 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster1 2023-08-26T13:59:16.576 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T13:59:16.576 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=image_47 2023-08-26T13:59:16.576 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local log=/tmp/tmp.TfDn6BcQ3N/cluster1-mirror-image_47.status 2023-08-26T13:59:16.576 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_47 --format xml 2023-08-26T13:59:16.577 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-08-26T13:59:16.622 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ true 2023-08-26T13:59:16.623 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-08-26T13:59:16.624 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:-:1.1: Document is empty 2023-08-26T13:59:16.624 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr: 2023-08-26T13:59:16.624 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:^ 2023-08-26T13:59:16.625 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local primary_snapshot_id= 2023-08-26T13:59:16.625 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ test 211 = '' 2023-08-26T13:59:16.626 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-08-26T13:59:16.626 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ sleep 0.4 2023-08-26T13:59:17.026 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_47 /tmp/tmp.TfDn6BcQ3N/cluster1-mirror-image_47.status 2023-08-26T13:59:17.026 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster1 2023-08-26T13:59:17.026 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T13:59:17.027 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=image_47 2023-08-26T13:59:17.027 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local log=/tmp/tmp.TfDn6BcQ3N/cluster1-mirror-image_47.status 2023-08-26T13:59:17.027 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_47 --format xml 2023-08-26T13:59:17.027 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-08-26T13:59:17.076 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ true 2023-08-26T13:59:17.077 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-08-26T13:59:17.078 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:-:1.1: Document is empty 2023-08-26T13:59:17.078 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr: 2023-08-26T13:59:17.078 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:^ 2023-08-26T13:59:17.079 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local primary_snapshot_id= 2023-08-26T13:59:17.079 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ test 211 = '' 2023-08-26T13:59:17.079 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-08-26T13:59:17.079 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ sleep 0.8 2023-08-26T13:59:17.399 INFO:tasks.ceph.cluster2.osd.0.smithi124.stderr:2023-08-26T13:59:17.398+0000 7fb60dfa7700 -1 reset not still connected to 0x5645518fe5b0 2023-08-26T13:59:17.400 INFO:tasks.ceph.cluster2.osd.0.smithi124.stderr:2023-08-26T13:59:17.399+0000 7fb60dfa7700 -1 reset not still connected to 0x56455eaec9c0 2023-08-26T13:59:17.880 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_47 /tmp/tmp.TfDn6BcQ3N/cluster1-mirror-image_47.status 2023-08-26T13:59:17.881 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster1 2023-08-26T13:59:17.881 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T13:59:17.881 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=image_47 2023-08-26T13:59:17.881 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local log=/tmp/tmp.TfDn6BcQ3N/cluster1-mirror-image_47.status 2023-08-26T13:59:17.881 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_47 --format xml 2023-08-26T13:59:17.881 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-08-26T13:59:17.924 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-08-26T13:59:17.926 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local primary_snapshot_id=211 2023-08-26T13:59:17.927 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ test 211 = 211 2023-08-26T13:59:17.927 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ return 0 2023-08-26T13:59:17.927 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ wait_for_snap_present cluster1 mirror image_47 snap 2023-08-26T13:59:17.927 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster1 2023-08-26T13:59:17.928 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T13:59:17.928 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=image_47 2023-08-26T13:59:17.928 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local snap_name=snap 2023-08-26T13:59:17.928 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local s 2023-08-26T13:59:17.928 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2023-08-26T13:59:17.929 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ sleep 1 2023-08-26T13:59:18.927 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rbd --cluster cluster1 info mirror/image_47@snap 2023-08-26T13:59:18.972 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:rbd image 'image_47': 2023-08-26T13:59:18.973 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: size 128 MiB in 32 objects 2023-08-26T13:59:18.973 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: order 22 (4 MiB objects) 2023-08-26T13:59:18.973 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: snapshot_count: 2 2023-08-26T13:59:18.973 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: id: 12c0c09ccc55 2023-08-26T13:59:18.974 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: block_name_prefix: rbd_data.12c0c09ccc55 2023-08-26T13:59:18.974 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: format: 2 2023-08-26T13:59:18.974 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: features: layering, exclusive-lock, non-primary 2023-08-26T13:59:18.974 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: op_features: 2023-08-26T13:59:18.974 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: flags: 2023-08-26T13:59:18.975 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: create_timestamp: Sat Aug 26 13:50:36 2023 2023-08-26T13:59:18.975 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: access_timestamp: Sat Aug 26 13:50:36 2023 2023-08-26T13:59:18.975 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: modify_timestamp: Sat Aug 26 13:50:36 2023 2023-08-26T13:59:18.975 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: protected: False 2023-08-26T13:59:18.975 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: mirroring state: enabled 2023-08-26T13:59:18.975 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: mirroring mode: snapshot 2023-08-26T13:59:18.976 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: mirroring global id: 530e7abb-9a3e-4245-a312-8ee5a47a865b 2023-08-26T13:59:18.976 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: mirroring primary: false 2023-08-26T13:59:18.977 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ return 0 2023-08-26T13:59:18.977 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ compare_image_snaps mirror image_47 snap 2023-08-26T13:59:18.978 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T13:59:18.978 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=image_47 2023-08-26T13:59:18.978 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local snap_name=snap 2023-08-26T13:59:18.978 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local ret=0 2023-08-26T13:59:18.978 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local rmt_export=/tmp/tmp.TfDn6BcQ3N/cluster2-mirror-image_47.export 2023-08-26T13:59:18.979 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local loc_export=/tmp/tmp.TfDn6BcQ3N/cluster1-mirror-image_47.export 2023-08-26T13:59:18.979 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rm -f /tmp/tmp.TfDn6BcQ3N/cluster2-mirror-image_47.export /tmp/tmp.TfDn6BcQ3N/cluster1-mirror-image_47.export 2023-08-26T13:59:18.979 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rbd --cluster cluster2 -p mirror export image_47@snap /tmp/tmp.TfDn6BcQ3N/cluster2-mirror-image_47.export 2023-08-26T13:59:19.154 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-08-26T13:59:19.160 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rbd --cluster cluster1 -p mirror export image_47@snap /tmp/tmp.TfDn6BcQ3N/cluster1-mirror-image_47.export 2023-08-26T13:59:19.388 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-08-26T13:59:19.395 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ cmp /tmp/tmp.TfDn6BcQ3N/cluster2-mirror-image_47.export /tmp/tmp.TfDn6BcQ3N/cluster1-mirror-image_47.export 2023-08-26T13:59:19.450 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rm -f /tmp/tmp.TfDn6BcQ3N/cluster2-mirror-image_47.export /tmp/tmp.TfDn6BcQ3N/cluster1-mirror-image_47.export 2023-08-26T13:59:19.487 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ return 0 2023-08-26T13:59:19.488 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-08-26T13:59:19.488 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ image=image_48 2023-08-26T13:59:19.488 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ create_snap cluster2 mirror image_48 snap 2023-08-26T13:59:19.488 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster2 2023-08-26T13:59:19.488 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T13:59:19.489 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=image_48 2023-08-26T13:59:19.489 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local snap_name=snap 2023-08-26T13:59:19.489 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rbd --cluster cluster2 -p mirror snap create image_48@snap --debug-rbd=20 --debug-journaler=20 2023-08-26T13:59:21.327 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ wait_for_image_replay_started cluster1 mirror image_48 2023-08-26T13:59:21.327 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster1 2023-08-26T13:59:21.327 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T13:59:21.328 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=image_48 2023-08-26T13:59:21.328 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ wait_for_image_replay_state cluster1 mirror image_48 started 2023-08-26T13:59:21.328 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster1 2023-08-26T13:59:21.328 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T13:59:21.328 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=image_48 2023-08-26T13:59:21.329 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local state=started 2023-08-26T13:59:21.329 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local s 2023-08-26T13:59:21.329 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-08-26T13:59:21.329 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ sleep 1 2023-08-26T13:59:22.328 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ test_image_replay_state cluster1 mirror image_48 started 2023-08-26T13:59:22.329 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster1 2023-08-26T13:59:22.329 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T13:59:22.329 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=image_48 2023-08-26T13:59:22.329 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local test_state=started 2023-08-26T13:59:22.329 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local status_result 2023-08-26T13:59:22.330 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local current_state=stopped 2023-08-26T13:59:22.332 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_48 2023-08-26T13:59:22.333 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ grep -i state 2023-08-26T13:59:22.333 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local cluster_instance=cluster1 2023-08-26T13:59:22.333 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ shift 2023-08-26T13:59:22.333 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local cluster=cluster1 2023-08-26T13:59:22.334 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local instance=cluster1 2023-08-26T13:59:22.334 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local loop_instance 2023-08-26T13:59:22.334 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-08-26T13:59:22.334 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ sleep 0 2023-08-26T13:59:22.334 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ '[' cluster1 '!=' cluster1 ']' 2023-08-26T13:59:22.334 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ seq 0 3 2023-08-26T13:59:22.335 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-08-26T13:59:22.335 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_48 2023-08-26T13:59:22.336 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local cluster=cluster1:0 2023-08-26T13:59:22.336 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ shift 2023-08-26T13:59:22.336 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local instance 2023-08-26T13:59:22.336 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ set_cluster_instance cluster1:0 cluster instance 2023-08-26T13:59:22.337 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local val=cluster1:0 2023-08-26T13:59:22.337 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local cluster_var_name=cluster 2023-08-26T13:59:22.337 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local instance_var_name=instance 2023-08-26T13:59:22.337 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ cluster=cluster1 2023-08-26T13:59:22.337 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ instance=0 2023-08-26T13:59:22.337 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ '[' 0 = cluster1:0 ']' 2023-08-26T13:59:22.338 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ eval cluster=cluster1 2023-08-26T13:59:22.338 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ cluster=cluster1 2023-08-26T13:59:22.338 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ eval instance=0 2023-08-26T13:59:22.338 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ instance=0 2023-08-26T13:59:22.338 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ daemon_asok_file cluster1:0 cluster1 2023-08-26T13:59:22.339 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ local local_cluster=cluster1:0 2023-08-26T13:59:22.339 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ local cluster=cluster1 2023-08-26T13:59:22.339 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ local instance 2023-08-26T13:59:22.340 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2023-08-26T13:59:22.341 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ local val=cluster1:0 2023-08-26T13:59:22.341 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ local cluster_var_name=local_cluster 2023-08-26T13:59:22.341 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ local instance_var_name=instance 2023-08-26T13:59:22.341 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ cluster=cluster1 2023-08-26T13:59:22.341 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ instance=0 2023-08-26T13:59:22.342 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ '[' 0 = cluster1:0 ']' 2023-08-26T13:59:22.342 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ eval local_cluster=cluster1 2023-08-26T13:59:22.342 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++++ local_cluster=cluster1 2023-08-26T13:59:22.342 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ eval instance=0 2023-08-26T13:59:22.342 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++++ instance=0 2023-08-26T13:59:22.342 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2023-08-26T13:59:22.355 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-08-26T13:59:22.355 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-08-26T13:59:22.355 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-08-26T13:59:22.356 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_48 2023-08-26T13:59:22.478 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ return 0 2023-08-26T13:59:22.478 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ status_result=' "state": "Replaying"' 2023-08-26T13:59:22.479 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ echo ' "state": "Replaying"' 2023-08-26T13:59:22.479 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ grep -i Replaying 2023-08-26T13:59:22.479 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: "state": "Replaying" 2023-08-26T13:59:22.481 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ current_state=started 2023-08-26T13:59:22.481 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ test started = started 2023-08-26T13:59:22.482 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ return 0 2023-08-26T13:59:22.482 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_48 2023-08-26T13:59:22.482 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local local_cluster=cluster1 2023-08-26T13:59:22.482 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster2 2023-08-26T13:59:22.482 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T13:59:22.483 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=image_48 2023-08-26T13:59:22.483 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ '[' snapshot = journal ']' 2023-08-26T13:59:22.483 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ '[' snapshot = snapshot ']' 2023-08-26T13:59:22.483 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_48 2023-08-26T13:59:22.483 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local local_cluster=cluster1 2023-08-26T13:59:22.484 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster2 2023-08-26T13:59:22.484 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T13:59:22.484 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=image_48 2023-08-26T13:59:22.484 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ mkfname cluster2-mirror-image_48.status 2023-08-26T13:59:22.484 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ echo cluster2-mirror-image_48.status 2023-08-26T13:59:22.484 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ sed -e 's|[/ ]|_|g' 2023-08-26T13:59:22.485 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local status_log=/tmp/tmp.TfDn6BcQ3N/cluster2-mirror-image_48.status 2023-08-26T13:59:22.486 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ mkfname cluster1-mirror-image_48.status 2023-08-26T13:59:22.486 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ echo cluster1-mirror-image_48.status 2023-08-26T13:59:22.486 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ sed -e 's|[/ ]|_|g' 2023-08-26T13:59:22.486 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local local_status_log=/tmp/tmp.TfDn6BcQ3N/cluster1-mirror-image_48.status 2023-08-26T13:59:22.487 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ mirror_image_snapshot cluster2 mirror image_48 2023-08-26T13:59:22.487 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster2 2023-08-26T13:59:22.487 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T13:59:22.487 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=image_48 2023-08-26T13:59:22.487 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_48 2023-08-26T13:59:23.332 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:Snapshot ID: 213 2023-08-26T13:59:23.342 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_48 /tmp/tmp.TfDn6BcQ3N/cluster2-mirror-image_48.status 2023-08-26T13:59:23.342 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster2 2023-08-26T13:59:23.342 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T13:59:23.342 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=image_48 2023-08-26T13:59:23.343 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local log=/tmp/tmp.TfDn6BcQ3N/cluster2-mirror-image_48.status 2023-08-26T13:59:23.343 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_48 --format xml 2023-08-26T13:59:23.343 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-08-26T13:59:23.383 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ xmlstarlet sel -t -v //snapshot/id 2023-08-26T13:59:23.385 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local snapshot_id=213 2023-08-26T13:59:23.385 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ true 2023-08-26T13:59:23.386 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-08-26T13:59:23.386 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ sleep 0.2 2023-08-26T13:59:23.586 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_48 /tmp/tmp.TfDn6BcQ3N/cluster1-mirror-image_48.status 2023-08-26T13:59:23.587 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster1 2023-08-26T13:59:23.587 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T13:59:23.587 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=image_48 2023-08-26T13:59:23.587 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local log=/tmp/tmp.TfDn6BcQ3N/cluster1-mirror-image_48.status 2023-08-26T13:59:23.587 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_48 --format xml 2023-08-26T13:59:23.588 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-08-26T13:59:23.632 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ true 2023-08-26T13:59:23.633 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-08-26T13:59:23.634 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:-:1.1: Document is empty 2023-08-26T13:59:23.634 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr: 2023-08-26T13:59:23.634 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:^ 2023-08-26T13:59:23.635 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local primary_snapshot_id= 2023-08-26T13:59:23.635 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ test 213 = '' 2023-08-26T13:59:23.635 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-08-26T13:59:23.635 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ sleep 0.4 2023-08-26T13:59:24.036 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_48 /tmp/tmp.TfDn6BcQ3N/cluster1-mirror-image_48.status 2023-08-26T13:59:24.036 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster1 2023-08-26T13:59:24.037 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T13:59:24.037 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=image_48 2023-08-26T13:59:24.037 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local log=/tmp/tmp.TfDn6BcQ3N/cluster1-mirror-image_48.status 2023-08-26T13:59:24.037 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_48 --format xml 2023-08-26T13:59:24.037 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-08-26T13:59:24.081 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ true 2023-08-26T13:59:24.081 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-08-26T13:59:24.082 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:-:1.1: Document is empty 2023-08-26T13:59:24.082 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr: 2023-08-26T13:59:24.083 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:^ 2023-08-26T13:59:24.083 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local primary_snapshot_id= 2023-08-26T13:59:24.083 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ test 213 = '' 2023-08-26T13:59:24.084 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-08-26T13:59:24.084 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ sleep 0.8 2023-08-26T13:59:24.884 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_48 /tmp/tmp.TfDn6BcQ3N/cluster1-mirror-image_48.status 2023-08-26T13:59:24.884 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster1 2023-08-26T13:59:24.885 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T13:59:24.885 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=image_48 2023-08-26T13:59:24.885 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local log=/tmp/tmp.TfDn6BcQ3N/cluster1-mirror-image_48.status 2023-08-26T13:59:24.885 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_48 --format xml 2023-08-26T13:59:24.886 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-08-26T13:59:24.933 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-08-26T13:59:24.935 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local primary_snapshot_id=213 2023-08-26T13:59:24.935 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ test 213 = 213 2023-08-26T13:59:24.936 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ return 0 2023-08-26T13:59:24.936 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ wait_for_snap_present cluster1 mirror image_48 snap 2023-08-26T13:59:24.936 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster1 2023-08-26T13:59:24.936 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T13:59:24.936 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=image_48 2023-08-26T13:59:24.937 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local snap_name=snap 2023-08-26T13:59:24.937 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local s 2023-08-26T13:59:24.937 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2023-08-26T13:59:24.937 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ sleep 1 2023-08-26T13:59:25.936 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rbd --cluster cluster1 info mirror/image_48@snap 2023-08-26T13:59:25.980 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:rbd image 'image_48': 2023-08-26T13:59:25.981 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: size 128 MiB in 32 objects 2023-08-26T13:59:25.981 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: order 22 (4 MiB objects) 2023-08-26T13:59:25.981 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: snapshot_count: 2 2023-08-26T13:59:25.982 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: id: 12cf9f9e0e19 2023-08-26T13:59:25.982 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: block_name_prefix: rbd_data.12cf9f9e0e19 2023-08-26T13:59:25.982 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: format: 2 2023-08-26T13:59:25.982 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: features: layering, exclusive-lock, non-primary 2023-08-26T13:59:25.983 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: op_features: 2023-08-26T13:59:25.983 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: flags: 2023-08-26T13:59:25.983 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: create_timestamp: Sat Aug 26 13:50:48 2023 2023-08-26T13:59:25.983 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: access_timestamp: Sat Aug 26 13:50:48 2023 2023-08-26T13:59:25.983 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: modify_timestamp: Sat Aug 26 13:50:48 2023 2023-08-26T13:59:25.984 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: protected: False 2023-08-26T13:59:25.984 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: mirroring state: enabled 2023-08-26T13:59:25.984 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: mirroring mode: snapshot 2023-08-26T13:59:25.984 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: mirroring global id: 6721bd5f-42ff-4eab-84c6-4ca33fff75c0 2023-08-26T13:59:25.984 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: mirroring primary: false 2023-08-26T13:59:25.986 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ return 0 2023-08-26T13:59:25.986 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ compare_image_snaps mirror image_48 snap 2023-08-26T13:59:25.986 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T13:59:25.986 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=image_48 2023-08-26T13:59:25.987 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local snap_name=snap 2023-08-26T13:59:25.987 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local ret=0 2023-08-26T13:59:25.987 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local rmt_export=/tmp/tmp.TfDn6BcQ3N/cluster2-mirror-image_48.export 2023-08-26T13:59:25.987 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local loc_export=/tmp/tmp.TfDn6BcQ3N/cluster1-mirror-image_48.export 2023-08-26T13:59:25.987 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rm -f /tmp/tmp.TfDn6BcQ3N/cluster2-mirror-image_48.export /tmp/tmp.TfDn6BcQ3N/cluster1-mirror-image_48.export 2023-08-26T13:59:25.988 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rbd --cluster cluster2 -p mirror export image_48@snap /tmp/tmp.TfDn6BcQ3N/cluster2-mirror-image_48.export 2023-08-26T13:59:26.163 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-08-26T13:59:26.169 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rbd --cluster cluster1 -p mirror export image_48@snap /tmp/tmp.TfDn6BcQ3N/cluster1-mirror-image_48.export 2023-08-26T13:59:26.411 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-08-26T13:59:26.418 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ cmp /tmp/tmp.TfDn6BcQ3N/cluster2-mirror-image_48.export /tmp/tmp.TfDn6BcQ3N/cluster1-mirror-image_48.export 2023-08-26T13:59:26.473 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rm -f /tmp/tmp.TfDn6BcQ3N/cluster2-mirror-image_48.export /tmp/tmp.TfDn6BcQ3N/cluster1-mirror-image_48.export 2023-08-26T13:59:26.511 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ return 0 2023-08-26T13:59:26.511 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-08-26T13:59:26.511 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ image=image_49 2023-08-26T13:59:26.512 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ create_snap cluster2 mirror image_49 snap 2023-08-26T13:59:26.512 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster2 2023-08-26T13:59:26.512 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T13:59:26.513 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=image_49 2023-08-26T13:59:26.513 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local snap_name=snap 2023-08-26T13:59:26.513 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rbd --cluster cluster2 -p mirror snap create image_49@snap --debug-rbd=20 --debug-journaler=20 2023-08-26T13:59:28.358 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ wait_for_image_replay_started cluster1 mirror image_49 2023-08-26T13:59:28.359 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster1 2023-08-26T13:59:28.359 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T13:59:28.359 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=image_49 2023-08-26T13:59:28.360 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ wait_for_image_replay_state cluster1 mirror image_49 started 2023-08-26T13:59:28.360 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster1 2023-08-26T13:59:28.360 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T13:59:28.360 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=image_49 2023-08-26T13:59:28.361 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local state=started 2023-08-26T13:59:28.361 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local s 2023-08-26T13:59:28.361 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-08-26T13:59:28.361 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ sleep 1 2023-08-26T13:59:28.766 INFO:tasks.rbd_mirror_thrash:kill cluster1.client.mirror.0 2023-08-26T13:59:28.766 INFO:tasks.rbd_mirror.cluster1.client.mirror.0:Sent signal 15 2023-08-26T13:59:28.767 INFO:tasks.rbd_mirror_thrash:waiting for 7 secs before reviving daemons 2023-08-26T13:59:28.767 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi124.stderr:2023-08-26T13:59:28.766+0000 7f5d0f7e4700 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.0 (PID: 128354) UID: 1000 2023-08-26T13:59:28.820 INFO:tasks.ceph.cluster2.osd.0.smithi124.stderr:2023-08-26T13:59:28.818+0000 7fb60dfa7700 -1 reset not still connected to 0x5645518fe5b0 2023-08-26T13:59:29.360 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ test_image_replay_state cluster1 mirror image_49 started 2023-08-26T13:59:29.360 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster1 2023-08-26T13:59:29.360 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T13:59:29.360 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=image_49 2023-08-26T13:59:29.360 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local test_state=started 2023-08-26T13:59:29.361 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local status_result 2023-08-26T13:59:29.361 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local current_state=stopped 2023-08-26T13:59:29.362 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_49 2023-08-26T13:59:29.362 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local cluster_instance=cluster1 2023-08-26T13:59:29.362 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ grep -i state 2023-08-26T13:59:29.363 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ shift 2023-08-26T13:59:29.363 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local cluster=cluster1 2023-08-26T13:59:29.363 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local instance=cluster1 2023-08-26T13:59:29.363 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local loop_instance 2023-08-26T13:59:29.364 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-08-26T13:59:29.364 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ sleep 0 2023-08-26T13:59:29.364 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ '[' cluster1 '!=' cluster1 ']' 2023-08-26T13:59:29.364 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ seq 0 3 2023-08-26T13:59:29.367 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-08-26T13:59:29.367 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_49 2023-08-26T13:59:29.367 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local cluster=cluster1:0 2023-08-26T13:59:29.367 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ shift 2023-08-26T13:59:29.368 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local instance 2023-08-26T13:59:29.368 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ set_cluster_instance cluster1:0 cluster instance 2023-08-26T13:59:29.368 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local val=cluster1:0 2023-08-26T13:59:29.368 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local cluster_var_name=cluster 2023-08-26T13:59:29.368 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local instance_var_name=instance 2023-08-26T13:59:29.368 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ cluster=cluster1 2023-08-26T13:59:29.369 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ instance=0 2023-08-26T13:59:29.369 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ '[' 0 = cluster1:0 ']' 2023-08-26T13:59:29.369 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ eval cluster=cluster1 2023-08-26T13:59:29.369 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ cluster=cluster1 2023-08-26T13:59:29.369 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ eval instance=0 2023-08-26T13:59:29.370 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ instance=0 2023-08-26T13:59:29.370 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ daemon_asok_file cluster1:0 cluster1 2023-08-26T13:59:29.370 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ local local_cluster=cluster1:0 2023-08-26T13:59:29.370 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ local cluster=cluster1 2023-08-26T13:59:29.370 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ local instance 2023-08-26T13:59:29.371 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2023-08-26T13:59:29.371 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ local val=cluster1:0 2023-08-26T13:59:29.371 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ local cluster_var_name=local_cluster 2023-08-26T13:59:29.371 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ local instance_var_name=instance 2023-08-26T13:59:29.371 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ cluster=cluster1 2023-08-26T13:59:29.372 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ instance=0 2023-08-26T13:59:29.372 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ '[' 0 = cluster1:0 ']' 2023-08-26T13:59:29.372 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ eval local_cluster=cluster1 2023-08-26T13:59:29.372 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++++ local_cluster=cluster1 2023-08-26T13:59:29.372 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ eval instance=0 2023-08-26T13:59:29.373 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++++ instance=0 2023-08-26T13:59:29.373 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2023-08-26T13:59:29.386 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-08-26T13:59:29.386 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-08-26T13:59:29.386 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-08-26T13:59:29.386 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_49 2023-08-26T13:59:29.475 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:admin_socket: exception getting command descriptions: [Errno 2] No such file or directory 2023-08-26T13:59:29.478 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-08-26T13:59:29.478 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_49 2023-08-26T13:59:29.478 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local cluster=cluster1:1 2023-08-26T13:59:29.479 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ shift 2023-08-26T13:59:29.479 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local instance 2023-08-26T13:59:29.479 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ set_cluster_instance cluster1:1 cluster instance 2023-08-26T13:59:29.479 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local val=cluster1:1 2023-08-26T13:59:29.479 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local cluster_var_name=cluster 2023-08-26T13:59:29.480 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local instance_var_name=instance 2023-08-26T13:59:29.480 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ cluster=cluster1 2023-08-26T13:59:29.480 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ instance=1 2023-08-26T13:59:29.480 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ '[' 1 = cluster1:1 ']' 2023-08-26T13:59:29.480 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ eval cluster=cluster1 2023-08-26T13:59:29.481 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ cluster=cluster1 2023-08-26T13:59:29.481 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ eval instance=1 2023-08-26T13:59:29.481 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ instance=1 2023-08-26T13:59:29.481 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ daemon_asok_file cluster1:1 cluster1 2023-08-26T13:59:29.481 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ local local_cluster=cluster1:1 2023-08-26T13:59:29.481 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ local cluster=cluster1 2023-08-26T13:59:29.482 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ local instance 2023-08-26T13:59:29.482 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2023-08-26T13:59:29.482 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ local val=cluster1:1 2023-08-26T13:59:29.482 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ local cluster_var_name=local_cluster 2023-08-26T13:59:29.482 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ local instance_var_name=instance 2023-08-26T13:59:29.483 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ cluster=cluster1 2023-08-26T13:59:29.483 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ instance=1 2023-08-26T13:59:29.483 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ '[' 1 = cluster1:1 ']' 2023-08-26T13:59:29.483 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ eval local_cluster=cluster1 2023-08-26T13:59:29.483 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++++ local_cluster=cluster1 2023-08-26T13:59:29.484 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ eval instance=1 2023-08-26T13:59:29.484 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++++ instance=1 2023-08-26T13:59:29.484 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2023-08-26T13:59:29.499 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-08-26T13:59:29.499 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-08-26T13:59:29.499 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-08-26T13:59:29.500 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_49 2023-08-26T13:59:29.612 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ return 0 2023-08-26T13:59:29.613 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ status_result=' "state": "Replaying"' 2023-08-26T13:59:29.613 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ echo ' "state": "Replaying"' 2023-08-26T13:59:29.613 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ grep -i Replaying 2023-08-26T13:59:29.614 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: "state": "Replaying" 2023-08-26T13:59:29.617 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ current_state=started 2023-08-26T13:59:29.617 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ test started = started 2023-08-26T13:59:29.617 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ return 0 2023-08-26T13:59:29.617 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_49 2023-08-26T13:59:29.617 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local local_cluster=cluster1 2023-08-26T13:59:29.618 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster2 2023-08-26T13:59:29.618 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T13:59:29.618 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=image_49 2023-08-26T13:59:29.618 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ '[' snapshot = journal ']' 2023-08-26T13:59:29.619 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ '[' snapshot = snapshot ']' 2023-08-26T13:59:29.619 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_49 2023-08-26T13:59:29.619 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local local_cluster=cluster1 2023-08-26T13:59:29.619 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster2 2023-08-26T13:59:29.619 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T13:59:29.620 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=image_49 2023-08-26T13:59:29.620 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ mkfname cluster2-mirror-image_49.status 2023-08-26T13:59:29.620 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ echo cluster2-mirror-image_49.status 2023-08-26T13:59:29.620 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ sed -e 's|[/ ]|_|g' 2023-08-26T13:59:29.621 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local status_log=/tmp/tmp.TfDn6BcQ3N/cluster2-mirror-image_49.status 2023-08-26T13:59:29.621 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ mkfname cluster1-mirror-image_49.status 2023-08-26T13:59:29.621 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ echo cluster1-mirror-image_49.status 2023-08-26T13:59:29.621 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ sed -e 's|[/ ]|_|g' 2023-08-26T13:59:29.622 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local local_status_log=/tmp/tmp.TfDn6BcQ3N/cluster1-mirror-image_49.status 2023-08-26T13:59:29.623 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ mirror_image_snapshot cluster2 mirror image_49 2023-08-26T13:59:29.623 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster2 2023-08-26T13:59:29.623 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T13:59:29.623 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=image_49 2023-08-26T13:59:29.623 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_49 2023-08-26T13:59:30.360 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:Snapshot ID: 215 2023-08-26T13:59:30.368 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_49 /tmp/tmp.TfDn6BcQ3N/cluster2-mirror-image_49.status 2023-08-26T13:59:30.369 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster2 2023-08-26T13:59:30.369 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T13:59:30.369 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=image_49 2023-08-26T13:59:30.369 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local log=/tmp/tmp.TfDn6BcQ3N/cluster2-mirror-image_49.status 2023-08-26T13:59:30.369 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_49 --format xml 2023-08-26T13:59:30.370 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-08-26T13:59:30.412 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ xmlstarlet sel -t -v //snapshot/id 2023-08-26T13:59:30.414 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local snapshot_id=215 2023-08-26T13:59:30.414 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ true 2023-08-26T13:59:30.414 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-08-26T13:59:30.415 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ sleep 0.2 2023-08-26T13:59:30.615 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_49 /tmp/tmp.TfDn6BcQ3N/cluster1-mirror-image_49.status 2023-08-26T13:59:30.616 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster1 2023-08-26T13:59:30.616 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T13:59:30.616 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=image_49 2023-08-26T13:59:30.616 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local log=/tmp/tmp.TfDn6BcQ3N/cluster1-mirror-image_49.status 2023-08-26T13:59:30.616 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_49 --format xml 2023-08-26T13:59:30.617 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-08-26T13:59:30.662 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-08-26T13:59:30.664 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local primary_snapshot_id=116 2023-08-26T13:59:30.664 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ test 215 = 116 2023-08-26T13:59:30.664 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-08-26T13:59:30.664 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ sleep 0.4 2023-08-26T13:59:31.066 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_49 /tmp/tmp.TfDn6BcQ3N/cluster1-mirror-image_49.status 2023-08-26T13:59:31.066 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster1 2023-08-26T13:59:31.066 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T13:59:31.067 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=image_49 2023-08-26T13:59:31.067 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local log=/tmp/tmp.TfDn6BcQ3N/cluster1-mirror-image_49.status 2023-08-26T13:59:31.067 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_49 --format xml 2023-08-26T13:59:31.068 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-08-26T13:59:31.113 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-08-26T13:59:31.114 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local primary_snapshot_id=116 2023-08-26T13:59:31.114 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ test 215 = 116 2023-08-26T13:59:31.115 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-08-26T13:59:31.115 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ sleep 0.8 2023-08-26T13:59:31.916 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_49 /tmp/tmp.TfDn6BcQ3N/cluster1-mirror-image_49.status 2023-08-26T13:59:31.916 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster1 2023-08-26T13:59:31.916 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T13:59:31.917 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=image_49 2023-08-26T13:59:31.917 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local log=/tmp/tmp.TfDn6BcQ3N/cluster1-mirror-image_49.status 2023-08-26T13:59:31.917 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_49 --format xml 2023-08-26T13:59:31.917 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-08-26T13:59:31.967 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ true 2023-08-26T13:59:31.967 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-08-26T13:59:31.968 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:-:1.1: Document is empty 2023-08-26T13:59:31.969 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr: 2023-08-26T13:59:31.969 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:^ 2023-08-26T13:59:31.969 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local primary_snapshot_id= 2023-08-26T13:59:31.970 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ test 215 = '' 2023-08-26T13:59:31.970 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-08-26T13:59:31.970 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ sleep 1.6 2023-08-26T13:59:33.570 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_49 /tmp/tmp.TfDn6BcQ3N/cluster1-mirror-image_49.status 2023-08-26T13:59:33.571 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster1 2023-08-26T13:59:33.571 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T13:59:33.571 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=image_49 2023-08-26T13:59:33.571 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local log=/tmp/tmp.TfDn6BcQ3N/cluster1-mirror-image_49.status 2023-08-26T13:59:33.572 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_49 --format xml 2023-08-26T13:59:33.572 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-08-26T13:59:33.621 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-08-26T13:59:33.623 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local primary_snapshot_id=215 2023-08-26T13:59:33.624 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ test 215 = 215 2023-08-26T13:59:33.624 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ return 0 2023-08-26T13:59:33.624 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ wait_for_snap_present cluster1 mirror image_49 snap 2023-08-26T13:59:33.624 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster1 2023-08-26T13:59:33.624 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T13:59:33.625 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=image_49 2023-08-26T13:59:33.625 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local snap_name=snap 2023-08-26T13:59:33.625 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local s 2023-08-26T13:59:33.625 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2023-08-26T13:59:33.625 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ sleep 1 2023-08-26T13:59:34.624 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rbd --cluster cluster1 info mirror/image_49@snap 2023-08-26T13:59:34.672 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:rbd image 'image_49': 2023-08-26T13:59:34.673 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: size 128 MiB in 32 objects 2023-08-26T13:59:34.673 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: order 22 (4 MiB objects) 2023-08-26T13:59:34.673 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: snapshot_count: 2 2023-08-26T13:59:34.673 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: id: 10b7e3b24b97 2023-08-26T13:59:34.673 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: block_name_prefix: rbd_data.10b7e3b24b97 2023-08-26T13:59:34.674 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: format: 2 2023-08-26T13:59:34.674 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: features: layering, exclusive-lock, non-primary 2023-08-26T13:59:34.674 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: op_features: 2023-08-26T13:59:34.674 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: flags: 2023-08-26T13:59:34.674 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: create_timestamp: Sat Aug 26 13:50:57 2023 2023-08-26T13:59:34.675 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: access_timestamp: Sat Aug 26 13:50:57 2023 2023-08-26T13:59:34.675 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: modify_timestamp: Sat Aug 26 13:50:57 2023 2023-08-26T13:59:34.675 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: protected: False 2023-08-26T13:59:34.675 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: mirroring state: enabled 2023-08-26T13:59:34.675 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: mirroring mode: snapshot 2023-08-26T13:59:34.676 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: mirroring global id: 5f1ee9ae-1ac1-4156-b437-c09951a7762e 2023-08-26T13:59:34.676 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: mirroring primary: false 2023-08-26T13:59:34.677 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ return 0 2023-08-26T13:59:34.677 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ compare_image_snaps mirror image_49 snap 2023-08-26T13:59:34.677 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T13:59:34.678 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=image_49 2023-08-26T13:59:34.678 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local snap_name=snap 2023-08-26T13:59:34.678 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local ret=0 2023-08-26T13:59:34.678 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local rmt_export=/tmp/tmp.TfDn6BcQ3N/cluster2-mirror-image_49.export 2023-08-26T13:59:34.678 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local loc_export=/tmp/tmp.TfDn6BcQ3N/cluster1-mirror-image_49.export 2023-08-26T13:59:34.679 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rm -f /tmp/tmp.TfDn6BcQ3N/cluster2-mirror-image_49.export /tmp/tmp.TfDn6BcQ3N/cluster1-mirror-image_49.export 2023-08-26T13:59:34.679 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rbd --cluster cluster2 -p mirror export image_49@snap /tmp/tmp.TfDn6BcQ3N/cluster2-mirror-image_49.export 2023-08-26T13:59:34.854 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-08-26T13:59:34.861 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rbd --cluster cluster1 -p mirror export image_49@snap /tmp/tmp.TfDn6BcQ3N/cluster1-mirror-image_49.export 2023-08-26T13:59:35.101 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-08-26T13:59:35.108 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ cmp /tmp/tmp.TfDn6BcQ3N/cluster2-mirror-image_49.export /tmp/tmp.TfDn6BcQ3N/cluster1-mirror-image_49.export 2023-08-26T13:59:35.163 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rm -f /tmp/tmp.TfDn6BcQ3N/cluster2-mirror-image_49.export /tmp/tmp.TfDn6BcQ3N/cluster1-mirror-image_49.export 2023-08-26T13:59:35.200 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ return 0 2023-08-26T13:59:35.201 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-08-26T13:59:35.201 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ image=image_50 2023-08-26T13:59:35.201 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ create_snap cluster2 mirror image_50 snap 2023-08-26T13:59:35.201 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster2 2023-08-26T13:59:35.201 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T13:59:35.202 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=image_50 2023-08-26T13:59:35.202 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local snap_name=snap 2023-08-26T13:59:35.202 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rbd --cluster cluster2 -p mirror snap create image_50@snap --debug-rbd=20 --debug-journaler=20 2023-08-26T13:59:35.768 INFO:tasks.rbd_mirror_thrash:waiting for cluster1.client.mirror.0 2023-08-26T13:59:35.768 INFO:teuthology.orchestra.run:waiting for 600 2023-08-26T13:59:35.768 INFO:tasks.rbd_mirror_thrash:reviving cluster1.client.mirror.0 2023-08-26T13:59:35.769 INFO:tasks.rbd_mirror.cluster1.client.mirror.0:Restarting daemon 2023-08-26T13:59:35.769 DEBUG:teuthology.orchestra.run.smithi124:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.0 2023-08-26T13:59:35.772 INFO:tasks.rbd_mirror.cluster1.client.mirror.0:Started 2023-08-26T13:59:35.772 INFO:tasks.rbd_mirror_thrash:waiting for 96 secs before thrashing 2023-08-26T13:59:37.230 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ wait_for_image_replay_started cluster1 mirror image_50 2023-08-26T13:59:37.230 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster1 2023-08-26T13:59:37.231 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T13:59:37.231 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=image_50 2023-08-26T13:59:37.231 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ wait_for_image_replay_state cluster1 mirror image_50 started 2023-08-26T13:59:37.231 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster1 2023-08-26T13:59:37.231 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T13:59:37.232 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=image_50 2023-08-26T13:59:37.232 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local state=started 2023-08-26T13:59:37.232 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local s 2023-08-26T13:59:37.232 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-08-26T13:59:37.232 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ sleep 1 2023-08-26T13:59:38.231 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ test_image_replay_state cluster1 mirror image_50 started 2023-08-26T13:59:38.232 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster1 2023-08-26T13:59:38.232 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T13:59:38.232 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=image_50 2023-08-26T13:59:38.232 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local test_state=started 2023-08-26T13:59:38.233 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local status_result 2023-08-26T13:59:38.233 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local current_state=stopped 2023-08-26T13:59:38.234 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_50 2023-08-26T13:59:38.234 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local cluster_instance=cluster1 2023-08-26T13:59:38.234 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ shift 2023-08-26T13:59:38.235 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ grep -i state 2023-08-26T13:59:38.235 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local cluster=cluster1 2023-08-26T13:59:38.235 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local instance=cluster1 2023-08-26T13:59:38.235 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local loop_instance 2023-08-26T13:59:38.236 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2023-08-26T13:59:38.236 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ sleep 0 2023-08-26T13:59:38.236 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ '[' cluster1 '!=' cluster1 ']' 2023-08-26T13:59:38.236 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ seq 0 3 2023-08-26T13:59:38.239 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-08-26T13:59:38.239 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_50 2023-08-26T13:59:38.239 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local cluster=cluster1:0 2023-08-26T13:59:38.239 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ shift 2023-08-26T13:59:38.240 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local instance 2023-08-26T13:59:38.240 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ set_cluster_instance cluster1:0 cluster instance 2023-08-26T13:59:38.240 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local val=cluster1:0 2023-08-26T13:59:38.240 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local cluster_var_name=cluster 2023-08-26T13:59:38.240 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local instance_var_name=instance 2023-08-26T13:59:38.240 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ cluster=cluster1 2023-08-26T13:59:38.241 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ instance=0 2023-08-26T13:59:38.241 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ '[' 0 = cluster1:0 ']' 2023-08-26T13:59:38.241 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ eval cluster=cluster1 2023-08-26T13:59:38.241 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ cluster=cluster1 2023-08-26T13:59:38.241 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ eval instance=0 2023-08-26T13:59:38.241 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ instance=0 2023-08-26T13:59:38.242 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ daemon_asok_file cluster1:0 cluster1 2023-08-26T13:59:38.242 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ local local_cluster=cluster1:0 2023-08-26T13:59:38.242 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ local cluster=cluster1 2023-08-26T13:59:38.242 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ local instance 2023-08-26T13:59:38.243 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2023-08-26T13:59:38.243 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ local val=cluster1:0 2023-08-26T13:59:38.243 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ local cluster_var_name=local_cluster 2023-08-26T13:59:38.243 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ local instance_var_name=instance 2023-08-26T13:59:38.244 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ cluster=cluster1 2023-08-26T13:59:38.244 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ instance=0 2023-08-26T13:59:38.244 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ '[' 0 = cluster1:0 ']' 2023-08-26T13:59:38.244 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ eval local_cluster=cluster1 2023-08-26T13:59:38.244 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++++ local_cluster=cluster1 2023-08-26T13:59:38.245 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ eval instance=0 2023-08-26T13:59:38.245 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++++ instance=0 2023-08-26T13:59:38.245 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2023-08-26T13:59:38.258 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-08-26T13:59:38.259 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-08-26T13:59:38.259 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-08-26T13:59:38.259 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_50 2023-08-26T13:59:38.360 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:Invalid command: unused arguments: ['mirror/image_50'] 2023-08-26T13:59:38.361 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:rbd mirror status : get status for rbd mirror 2023-08-26T13:59:38.361 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:admin_socket: invalid command 2023-08-26T13:59:38.364 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-08-26T13:59:38.364 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_50 2023-08-26T13:59:38.364 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local cluster=cluster1:1 2023-08-26T13:59:38.364 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ shift 2023-08-26T13:59:38.365 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local instance 2023-08-26T13:59:38.365 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ set_cluster_instance cluster1:1 cluster instance 2023-08-26T13:59:38.365 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local val=cluster1:1 2023-08-26T13:59:38.365 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local cluster_var_name=cluster 2023-08-26T13:59:38.365 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local instance_var_name=instance 2023-08-26T13:59:38.366 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ cluster=cluster1 2023-08-26T13:59:38.366 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ instance=1 2023-08-26T13:59:38.366 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ '[' 1 = cluster1:1 ']' 2023-08-26T13:59:38.366 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ eval cluster=cluster1 2023-08-26T13:59:38.366 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ cluster=cluster1 2023-08-26T13:59:38.366 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ eval instance=1 2023-08-26T13:59:38.367 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ instance=1 2023-08-26T13:59:38.367 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ daemon_asok_file cluster1:1 cluster1 2023-08-26T13:59:38.367 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ local local_cluster=cluster1:1 2023-08-26T13:59:38.367 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ local cluster=cluster1 2023-08-26T13:59:38.367 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ local instance 2023-08-26T13:59:38.368 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2023-08-26T13:59:38.368 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ local val=cluster1:1 2023-08-26T13:59:38.368 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ local cluster_var_name=local_cluster 2023-08-26T13:59:38.368 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ local instance_var_name=instance 2023-08-26T13:59:38.368 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ cluster=cluster1 2023-08-26T13:59:38.369 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ instance=1 2023-08-26T13:59:38.369 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ '[' 1 = cluster1:1 ']' 2023-08-26T13:59:38.369 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ eval local_cluster=cluster1 2023-08-26T13:59:38.369 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++++ local_cluster=cluster1 2023-08-26T13:59:38.369 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ eval instance=1 2023-08-26T13:59:38.370 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++++ instance=1 2023-08-26T13:59:38.370 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2023-08-26T13:59:38.384 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-08-26T13:59:38.384 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-08-26T13:59:38.384 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-08-26T13:59:38.385 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_50 2023-08-26T13:59:38.500 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ return 0 2023-08-26T13:59:38.501 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ status_result=' "state": "Replaying"' 2023-08-26T13:59:38.501 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ echo ' "state": "Replaying"' 2023-08-26T13:59:38.501 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ grep -i Replaying 2023-08-26T13:59:38.502 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: "state": "Replaying" 2023-08-26T13:59:38.504 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ current_state=started 2023-08-26T13:59:38.504 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ test started = started 2023-08-26T13:59:38.504 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ return 0 2023-08-26T13:59:38.504 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_50 2023-08-26T13:59:38.504 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local local_cluster=cluster1 2023-08-26T13:59:38.505 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster2 2023-08-26T13:59:38.505 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T13:59:38.505 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=image_50 2023-08-26T13:59:38.505 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ '[' snapshot = journal ']' 2023-08-26T13:59:38.505 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ '[' snapshot = snapshot ']' 2023-08-26T13:59:38.506 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_50 2023-08-26T13:59:38.506 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local local_cluster=cluster1 2023-08-26T13:59:38.506 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster2 2023-08-26T13:59:38.506 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T13:59:38.506 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=image_50 2023-08-26T13:59:38.507 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ mkfname cluster2-mirror-image_50.status 2023-08-26T13:59:38.507 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ echo cluster2-mirror-image_50.status 2023-08-26T13:59:38.507 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ sed -e 's|[/ ]|_|g' 2023-08-26T13:59:38.507 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local status_log=/tmp/tmp.TfDn6BcQ3N/cluster2-mirror-image_50.status 2023-08-26T13:59:38.507 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ mkfname cluster1-mirror-image_50.status 2023-08-26T13:59:38.507 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ echo cluster1-mirror-image_50.status 2023-08-26T13:59:38.508 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ sed -e 's|[/ ]|_|g' 2023-08-26T13:59:38.508 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local local_status_log=/tmp/tmp.TfDn6BcQ3N/cluster1-mirror-image_50.status 2023-08-26T13:59:38.509 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ mirror_image_snapshot cluster2 mirror image_50 2023-08-26T13:59:38.509 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster2 2023-08-26T13:59:38.509 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T13:59:38.509 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=image_50 2023-08-26T13:59:38.509 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_50 2023-08-26T13:59:39.230 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:Snapshot ID: 217 2023-08-26T13:59:39.240 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_50 /tmp/tmp.TfDn6BcQ3N/cluster2-mirror-image_50.status 2023-08-26T13:59:39.241 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster2 2023-08-26T13:59:39.241 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T13:59:39.241 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=image_50 2023-08-26T13:59:39.241 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local log=/tmp/tmp.TfDn6BcQ3N/cluster2-mirror-image_50.status 2023-08-26T13:59:39.242 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_50 --format xml 2023-08-26T13:59:39.242 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-08-26T13:59:39.285 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ xmlstarlet sel -t -v //snapshot/id 2023-08-26T13:59:39.286 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local snapshot_id=217 2023-08-26T13:59:39.287 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ true 2023-08-26T13:59:39.287 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-08-26T13:59:39.287 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ sleep 0.2 2023-08-26T13:59:39.488 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_50 /tmp/tmp.TfDn6BcQ3N/cluster1-mirror-image_50.status 2023-08-26T13:59:39.488 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster1 2023-08-26T13:59:39.489 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T13:59:39.489 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=image_50 2023-08-26T13:59:39.489 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local log=/tmp/tmp.TfDn6BcQ3N/cluster1-mirror-image_50.status 2023-08-26T13:59:39.489 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_50 --format xml 2023-08-26T13:59:39.489 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-08-26T13:59:39.535 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-08-26T13:59:39.537 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local primary_snapshot_id=117 2023-08-26T13:59:39.537 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ test 217 = 117 2023-08-26T13:59:39.537 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-08-26T13:59:39.538 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ sleep 0.4 2023-08-26T13:59:39.938 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_50 /tmp/tmp.TfDn6BcQ3N/cluster1-mirror-image_50.status 2023-08-26T13:59:39.938 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster1 2023-08-26T13:59:39.939 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T13:59:39.939 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=image_50 2023-08-26T13:59:39.939 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local log=/tmp/tmp.TfDn6BcQ3N/cluster1-mirror-image_50.status 2023-08-26T13:59:39.940 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_50 --format xml 2023-08-26T13:59:39.940 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-08-26T13:59:39.983 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ true 2023-08-26T13:59:39.984 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-08-26T13:59:39.985 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:-:1.1: Document is empty 2023-08-26T13:59:39.985 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr: 2023-08-26T13:59:39.985 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:^ 2023-08-26T13:59:39.986 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local primary_snapshot_id= 2023-08-26T13:59:39.986 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ test 217 = '' 2023-08-26T13:59:39.986 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2023-08-26T13:59:39.987 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ sleep 0.8 2023-08-26T13:59:40.444 INFO:tasks.ceph.cluster2.osd.2.smithi124.stderr:2023-08-26T13:59:40.442+0000 7fb790854700 -1 reset not still connected to 0x5598e1855e10 2023-08-26T13:59:40.445 INFO:tasks.ceph.cluster2.osd.2.smithi124.stderr:2023-08-26T13:59:40.442+0000 7fb790854700 -1 reset not still connected to 0x5598e427f110 2023-08-26T13:59:40.445 INFO:tasks.ceph.cluster2.osd.2.smithi124.stderr:2023-08-26T13:59:40.442+0000 7fb790854700 -1 reset not still connected to 0x5599107b9110 2023-08-26T13:59:40.787 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_50 /tmp/tmp.TfDn6BcQ3N/cluster1-mirror-image_50.status 2023-08-26T13:59:40.787 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster1 2023-08-26T13:59:40.787 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T13:59:40.787 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=image_50 2023-08-26T13:59:40.788 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local log=/tmp/tmp.TfDn6BcQ3N/cluster1-mirror-image_50.status 2023-08-26T13:59:40.788 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_50 --format xml 2023-08-26T13:59:40.788 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2023-08-26T13:59:40.831 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-08-26T13:59:40.834 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local primary_snapshot_id=217 2023-08-26T13:59:40.834 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ test 217 = 217 2023-08-26T13:59:40.835 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ return 0 2023-08-26T13:59:40.835 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ wait_for_snap_present cluster1 mirror image_50 snap 2023-08-26T13:59:40.835 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster1 2023-08-26T13:59:40.836 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T13:59:40.836 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=image_50 2023-08-26T13:59:40.836 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local snap_name=snap 2023-08-26T13:59:40.836 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local s 2023-08-26T13:59:40.837 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2023-08-26T13:59:40.837 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ sleep 1 2023-08-26T13:59:41.834 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rbd --cluster cluster1 info mirror/image_50@snap 2023-08-26T13:59:41.882 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:rbd image 'image_50': 2023-08-26T13:59:41.882 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: size 128 MiB in 32 objects 2023-08-26T13:59:41.882 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: order 22 (4 MiB objects) 2023-08-26T13:59:41.882 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: snapshot_count: 2 2023-08-26T13:59:41.883 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: id: 12e58da771c7 2023-08-26T13:59:41.883 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: block_name_prefix: rbd_data.12e58da771c7 2023-08-26T13:59:41.883 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: format: 2 2023-08-26T13:59:41.883 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: features: layering, exclusive-lock, non-primary 2023-08-26T13:59:41.883 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: op_features: 2023-08-26T13:59:41.884 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: flags: 2023-08-26T13:59:41.884 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: create_timestamp: Sat Aug 26 13:51:11 2023 2023-08-26T13:59:41.884 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: access_timestamp: Sat Aug 26 13:51:11 2023 2023-08-26T13:59:41.884 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: modify_timestamp: Sat Aug 26 13:51:11 2023 2023-08-26T13:59:41.884 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: protected: False 2023-08-26T13:59:41.885 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: mirroring state: enabled 2023-08-26T13:59:41.885 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: mirroring mode: snapshot 2023-08-26T13:59:41.885 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: mirroring global id: ebb4d874-359b-4be6-9915-df44e455c46f 2023-08-26T13:59:41.885 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: mirroring primary: false 2023-08-26T13:59:41.886 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ return 0 2023-08-26T13:59:41.887 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ compare_image_snaps mirror image_50 snap 2023-08-26T13:59:41.887 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T13:59:41.887 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=image_50 2023-08-26T13:59:41.887 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local snap_name=snap 2023-08-26T13:59:41.887 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local ret=0 2023-08-26T13:59:41.888 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local rmt_export=/tmp/tmp.TfDn6BcQ3N/cluster2-mirror-image_50.export 2023-08-26T13:59:41.888 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local loc_export=/tmp/tmp.TfDn6BcQ3N/cluster1-mirror-image_50.export 2023-08-26T13:59:41.888 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rm -f /tmp/tmp.TfDn6BcQ3N/cluster2-mirror-image_50.export /tmp/tmp.TfDn6BcQ3N/cluster1-mirror-image_50.export 2023-08-26T13:59:41.888 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rbd --cluster cluster2 -p mirror export image_50@snap /tmp/tmp.TfDn6BcQ3N/cluster2-mirror-image_50.export 2023-08-26T13:59:42.075 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-08-26T13:59:42.082 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rbd --cluster cluster1 -p mirror export image_50@snap /tmp/tmp.TfDn6BcQ3N/cluster1-mirror-image_50.export 2023-08-26T13:59:42.325 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-08-26T13:59:42.332 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ cmp /tmp/tmp.TfDn6BcQ3N/cluster2-mirror-image_50.export /tmp/tmp.TfDn6BcQ3N/cluster1-mirror-image_50.export 2023-08-26T13:59:42.387 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rm -f /tmp/tmp.TfDn6BcQ3N/cluster2-mirror-image_50.export /tmp/tmp.TfDn6BcQ3N/cluster1-mirror-image_50.export 2023-08-26T13:59:42.424 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ return 0 2023-08-26T13:59:42.424 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ testlog 'TEST: delete many images' 2023-08-26T13:59:42.424 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ tee -a /tmp/tmp.TfDn6BcQ3N/rbd-mirror.test.log 2023-08-26T13:59:42.425 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ date '+%F %T' 2023-08-26T13:59:42.425 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ echo 2023-08-26 13:59:42 TEST: delete many images 2023-08-26T13:59:42.426 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:2023-08-26 13:59:42 TEST: delete many images 2023-08-26T13:59:42.426 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ seq 1 50 2023-08-26T13:59:42.427 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-08-26T13:59:42.427 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ image=image_1 2023-08-26T13:59:42.427 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ remove_snapshot cluster2 mirror image_1 snap 2023-08-26T13:59:42.427 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster2 2023-08-26T13:59:42.428 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T13:59:42.428 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=image_1 2023-08-26T13:59:42.428 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local snap=snap 2023-08-26T13:59:42.428 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rbd --cluster cluster2 snap rm mirror/image_1@snap 2023-08-26T13:59:43.318 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr: Removing snap: 100% complete...done. 2023-08-26T13:59:43.332 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ remove_image_retry cluster2 mirror image_1 2023-08-26T13:59:43.332 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster2 2023-08-26T13:59:43.332 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T13:59:43.332 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=image_1 2023-08-26T13:59:43.333 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ for s in 0 1 2 4 8 16 32 2023-08-26T13:59:43.333 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ sleep 0 2023-08-26T13:59:43.334 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ remove_image cluster2 mirror image_1 2023-08-26T13:59:43.334 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster2 2023-08-26T13:59:43.334 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T13:59:43.334 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=image_1 2023-08-26T13:59:43.334 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_1 2023-08-26T13:59:43.424 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rbd --cluster=cluster2 rm mirror/image_1 2023-08-26T13:59:43.738 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi124.stderr:2023-08-26T13:59:43.736+0000 7fc244bc9700 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x555d5bf569c0 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2023-08-26T13:59:43.739 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi124.stderr:2023-08-26T13:59:43.736+0000 7fc24b3d6700 -1 rbd::mirror::ImageReplayer: 0x555d5ae9b180 [3/cab3ca63-f126-4865-b2a3-b025783f5e8a] operator(): start failed: (85) Interrupted system call should be restarted 2023-08-26T13:59:44.450 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2023-08-26T13:59:44.454 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ return 0 2023-08-26T13:59:44.454 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-08-26T13:59:44.454 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ image=image_2 2023-08-26T13:59:44.454 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ remove_snapshot cluster2 mirror image_2 snap 2023-08-26T13:59:44.454 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster2 2023-08-26T13:59:44.455 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T13:59:44.455 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=image_2 2023-08-26T13:59:44.455 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local snap=snap 2023-08-26T13:59:44.455 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rbd --cluster cluster2 snap rm mirror/image_2@snap 2023-08-26T13:59:45.318 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr: Removing snap: 100% complete...done. 2023-08-26T13:59:45.326 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ remove_image_retry cluster2 mirror image_2 2023-08-26T13:59:45.326 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster2 2023-08-26T13:59:45.326 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T13:59:45.326 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=image_2 2023-08-26T13:59:45.327 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ for s in 0 1 2 4 8 16 32 2023-08-26T13:59:45.327 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ sleep 0 2023-08-26T13:59:45.327 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ remove_image cluster2 mirror image_2 2023-08-26T13:59:45.328 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster2 2023-08-26T13:59:45.328 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T13:59:45.328 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=image_2 2023-08-26T13:59:45.328 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_2 2023-08-26T13:59:45.380 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rbd --cluster=cluster2 rm mirror/image_2 2023-08-26T13:59:45.688 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi124.stderr:2023-08-26T13:59:45.686+0000 7fc0c3135700 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x5567e0d511e0 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2023-08-26T13:59:45.688 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi124.stderr:2023-08-26T13:59:45.686+0000 7fc0c9942700 -1 rbd::mirror::ImageReplayer: 0x5567e030db80 [3/072383ad-81e1-4717-a595-674819d66895] operator(): start failed: (85) Interrupted system call should be restarted 2023-08-26T13:59:46.447 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2023-08-26T13:59:46.450 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ return 0 2023-08-26T13:59:46.450 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-08-26T13:59:46.450 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ image=image_3 2023-08-26T13:59:46.450 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ remove_snapshot cluster2 mirror image_3 snap 2023-08-26T13:59:46.451 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster2 2023-08-26T13:59:46.451 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T13:59:46.451 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=image_3 2023-08-26T13:59:46.451 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local snap=snap 2023-08-26T13:59:46.452 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rbd --cluster cluster2 snap rm mirror/image_3@snap 2023-08-26T13:59:47.339 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr: Removing snap: 100% complete...done. 2023-08-26T13:59:47.347 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ remove_image_retry cluster2 mirror image_3 2023-08-26T13:59:47.347 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster2 2023-08-26T13:59:47.347 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T13:59:47.348 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=image_3 2023-08-26T13:59:47.348 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ for s in 0 1 2 4 8 16 32 2023-08-26T13:59:47.348 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ sleep 0 2023-08-26T13:59:47.349 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ remove_image cluster2 mirror image_3 2023-08-26T13:59:47.349 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster2 2023-08-26T13:59:47.349 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T13:59:47.349 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=image_3 2023-08-26T13:59:47.350 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_3 2023-08-26T13:59:47.394 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rbd --cluster=cluster2 rm mirror/image_3 2023-08-26T13:59:47.499 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi124.stderr:2023-08-26T13:59:47.497+0000 7f6c3363a700 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x5621b9c0dba0 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2023-08-26T13:59:47.499 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi124.stderr:2023-08-26T13:59:47.497+0000 7f6c39646700 -1 rbd::mirror::ImageReplayer: 0x5621ba456280 [3/339416f3-5c35-427b-8a40-65d1d209d742] operator(): start failed: (85) Interrupted system call should be restarted 2023-08-26T13:59:47.620 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi124.stderr:2023-08-26T13:59:47.617+0000 7f6c3363a700 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x5621ba71ab60 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2023-08-26T13:59:47.620 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi124.stderr:2023-08-26T13:59:47.617+0000 7f6c39e47700 -1 rbd::mirror::ImageReplayer: 0x5621ba456280 [3/339416f3-5c35-427b-8a40-65d1d209d742] operator(): start failed: (85) Interrupted system call should be restarted 2023-08-26T13:59:48.461 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2023-08-26T13:59:48.465 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ return 0 2023-08-26T13:59:48.465 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-08-26T13:59:48.465 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ image=image_4 2023-08-26T13:59:48.465 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ remove_snapshot cluster2 mirror image_4 snap 2023-08-26T13:59:48.466 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster2 2023-08-26T13:59:48.466 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T13:59:48.466 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=image_4 2023-08-26T13:59:48.466 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local snap=snap 2023-08-26T13:59:48.466 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rbd --cluster cluster2 snap rm mirror/image_4@snap 2023-08-26T13:59:49.213 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr: Removing snap: 100% complete...done. 2023-08-26T13:59:49.225 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ remove_image_retry cluster2 mirror image_4 2023-08-26T13:59:49.225 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster2 2023-08-26T13:59:49.226 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T13:59:49.226 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=image_4 2023-08-26T13:59:49.226 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ for s in 0 1 2 4 8 16 32 2023-08-26T13:59:49.226 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ sleep 0 2023-08-26T13:59:49.227 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ remove_image cluster2 mirror image_4 2023-08-26T13:59:49.227 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster2 2023-08-26T13:59:49.227 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T13:59:49.228 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=image_4 2023-08-26T13:59:49.228 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_4 2023-08-26T13:59:49.273 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rbd --cluster=cluster2 rm mirror/image_4 2023-08-26T13:59:49.426 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi124.stderr:2023-08-26T13:59:49.424+0000 7fcd42daf700 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x563b71e91040 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2023-08-26T13:59:49.427 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi124.stderr:2023-08-26T13:59:49.424+0000 7fcd495bc700 -1 rbd::mirror::ImageReplayer: 0x563b71eeca00 [3/2f906edf-8e50-48f7-a42e-a4055f204c44] operator(): start failed: (85) Interrupted system call should be restarted 2023-08-26T13:59:50.471 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2023-08-26T13:59:50.474 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ return 0 2023-08-26T13:59:50.475 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-08-26T13:59:50.475 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ image=image_5 2023-08-26T13:59:50.475 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ remove_snapshot cluster2 mirror image_5 snap 2023-08-26T13:59:50.475 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster2 2023-08-26T13:59:50.476 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T13:59:50.476 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=image_5 2023-08-26T13:59:50.476 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local snap=snap 2023-08-26T13:59:50.476 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rbd --cluster cluster2 snap rm mirror/image_5@snap 2023-08-26T13:59:51.353 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr: Removing snap: 100% complete...done. 2023-08-26T13:59:51.364 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ remove_image_retry cluster2 mirror image_5 2023-08-26T13:59:51.364 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster2 2023-08-26T13:59:51.364 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T13:59:51.364 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=image_5 2023-08-26T13:59:51.365 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ for s in 0 1 2 4 8 16 32 2023-08-26T13:59:51.365 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ sleep 0 2023-08-26T13:59:51.366 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ remove_image cluster2 mirror image_5 2023-08-26T13:59:51.366 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster2 2023-08-26T13:59:51.366 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T13:59:51.366 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=image_5 2023-08-26T13:59:51.367 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_5 2023-08-26T13:59:51.412 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rbd --cluster=cluster2 rm mirror/image_5 2023-08-26T13:59:51.517 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi124.stderr:2023-08-26T13:59:51.515+0000 7fc244bc9700 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x555d5bf56b60 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2023-08-26T13:59:51.518 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi124.stderr:2023-08-26T13:59:51.515+0000 7fc24b3d6700 -1 rbd::mirror::ImageReplayer: 0x555d5b990500 [3/fd599976-dbcc-4e1b-98dd-2440bc289607] operator(): start failed: (85) Interrupted system call should be restarted 2023-08-26T13:59:52.483 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2023-08-26T13:59:52.486 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ return 0 2023-08-26T13:59:52.487 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-08-26T13:59:52.487 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ image=image_6 2023-08-26T13:59:52.487 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ remove_snapshot cluster2 mirror image_6 snap 2023-08-26T13:59:52.487 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster2 2023-08-26T13:59:52.487 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T13:59:52.488 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=image_6 2023-08-26T13:59:52.488 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local snap=snap 2023-08-26T13:59:52.488 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rbd --cluster cluster2 snap rm mirror/image_6@snap 2023-08-26T13:59:53.352 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr: Removing snap: 100% complete...done. 2023-08-26T13:59:53.361 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ remove_image_retry cluster2 mirror image_6 2023-08-26T13:59:53.362 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster2 2023-08-26T13:59:53.362 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T13:59:53.362 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=image_6 2023-08-26T13:59:53.362 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ for s in 0 1 2 4 8 16 32 2023-08-26T13:59:53.362 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ sleep 0 2023-08-26T13:59:53.364 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ remove_image cluster2 mirror image_6 2023-08-26T13:59:53.364 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster2 2023-08-26T13:59:53.364 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T13:59:53.365 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=image_6 2023-08-26T13:59:53.365 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_6 2023-08-26T13:59:53.413 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rbd --cluster=cluster2 rm mirror/image_6 2023-08-26T13:59:53.518 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi124.stderr:2023-08-26T13:59:53.516+0000 7fc0c3936700 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x5567e0b604e0 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2023-08-26T13:59:54.355 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2023-08-26T13:59:54.358 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ return 0 2023-08-26T13:59:54.359 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-08-26T13:59:54.359 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ image=image_7 2023-08-26T13:59:54.359 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ remove_snapshot cluster2 mirror image_7 snap 2023-08-26T13:59:54.359 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster2 2023-08-26T13:59:54.359 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T13:59:54.360 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=image_7 2023-08-26T13:59:54.360 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local snap=snap 2023-08-26T13:59:54.360 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rbd --cluster cluster2 snap rm mirror/image_7@snap 2023-08-26T13:59:55.347 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr: Removing snap: 100% complete...done. 2023-08-26T13:59:55.360 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ remove_image_retry cluster2 mirror image_7 2023-08-26T13:59:55.360 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster2 2023-08-26T13:59:55.360 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T13:59:55.361 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=image_7 2023-08-26T13:59:55.361 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ for s in 0 1 2 4 8 16 32 2023-08-26T13:59:55.361 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ sleep 0 2023-08-26T13:59:55.362 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ remove_image cluster2 mirror image_7 2023-08-26T13:59:55.362 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster2 2023-08-26T13:59:55.362 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T13:59:55.362 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=image_7 2023-08-26T13:59:55.362 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_7 2023-08-26T13:59:55.409 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rbd --cluster=cluster2 rm mirror/image_7 2023-08-26T13:59:55.474 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi124.stderr:2023-08-26T13:59:55.472+0000 7f0345350700 -1 rbd::mirror::ImageMap: 0x55798cef6ea0 handle_peer_ack_remove: failed to remove global_image_id=cdbf4c2c-a074-4ef2-b5df-d98b1449c167 2023-08-26T13:59:56.492 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2023-08-26T13:59:56.496 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ return 0 2023-08-26T13:59:56.496 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-08-26T13:59:56.496 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ image=image_8 2023-08-26T13:59:56.497 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ remove_snapshot cluster2 mirror image_8 snap 2023-08-26T13:59:56.497 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster2 2023-08-26T13:59:56.497 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T13:59:56.497 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=image_8 2023-08-26T13:59:56.497 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local snap=snap 2023-08-26T13:59:56.498 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rbd --cluster cluster2 snap rm mirror/image_8@snap 2023-08-26T13:59:57.361 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr: Removing snap: 100% complete...done. 2023-08-26T13:59:57.370 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ remove_image_retry cluster2 mirror image_8 2023-08-26T13:59:57.370 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster2 2023-08-26T13:59:57.370 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T13:59:57.370 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=image_8 2023-08-26T13:59:57.371 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ for s in 0 1 2 4 8 16 32 2023-08-26T13:59:57.371 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ sleep 0 2023-08-26T13:59:57.372 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ remove_image cluster2 mirror image_8 2023-08-26T13:59:57.372 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster2 2023-08-26T13:59:57.372 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T13:59:57.372 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=image_8 2023-08-26T13:59:57.372 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_8 2023-08-26T13:59:57.429 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rbd --cluster=cluster2 rm mirror/image_8 2023-08-26T13:59:57.541 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi124.stderr:2023-08-26T13:59:57.539+0000 7f6c3363a700 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x5621ba1db1e0 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2023-08-26T13:59:57.541 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi124.stderr:2023-08-26T13:59:57.539+0000 7f6c39e47700 -1 rbd::mirror::ImageReplayer: 0x5621ba456500 [3/0e8ffb4e-bbd1-41aa-b41a-6b2aebda2762] operator(): start failed: (85) Interrupted system call should be restarted 2023-08-26T13:59:58.492 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2023-08-26T13:59:58.496 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ return 0 2023-08-26T13:59:58.496 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-08-26T13:59:58.496 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ image=image_9 2023-08-26T13:59:58.497 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ remove_snapshot cluster2 mirror image_9 snap 2023-08-26T13:59:58.497 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster2 2023-08-26T13:59:58.497 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T13:59:58.497 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=image_9 2023-08-26T13:59:58.497 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local snap=snap 2023-08-26T13:59:58.498 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rbd --cluster cluster2 snap rm mirror/image_9@snap 2023-08-26T13:59:59.228 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr: Removing snap: 100% complete...done. 2023-08-26T13:59:59.239 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ remove_image_retry cluster2 mirror image_9 2023-08-26T13:59:59.239 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster2 2023-08-26T13:59:59.239 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T13:59:59.240 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=image_9 2023-08-26T13:59:59.240 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ for s in 0 1 2 4 8 16 32 2023-08-26T13:59:59.240 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ sleep 0 2023-08-26T13:59:59.241 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ remove_image cluster2 mirror image_9 2023-08-26T13:59:59.241 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster2 2023-08-26T13:59:59.241 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T13:59:59.241 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=image_9 2023-08-26T13:59:59.241 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_9 2023-08-26T13:59:59.284 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rbd --cluster=cluster2 rm mirror/image_9 2023-08-26T13:59:59.502 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi124.stderr:2023-08-26T13:59:59.501+0000 7fc2453ca700 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x555d5bebf040 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2023-08-26T13:59:59.502 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi124.stderr:2023-08-26T13:59:59.501+0000 7fc24bbd7700 -1 rbd::mirror::ImageReplayer: 0x555d5ae9bb80 [3/680b775a-fecc-478c-ab5f-9dc6f2f67724] operator(): start failed: (85) Interrupted system call should be restarted 2023-08-26T14:00:00.494 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2023-08-26T14:00:00.497 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ return 0 2023-08-26T14:00:00.497 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-08-26T14:00:00.498 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ image=image_10 2023-08-26T14:00:00.498 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ remove_snapshot cluster2 mirror image_10 snap 2023-08-26T14:00:00.498 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster2 2023-08-26T14:00:00.498 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T14:00:00.498 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=image_10 2023-08-26T14:00:00.498 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local snap=snap 2023-08-26T14:00:00.499 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rbd --cluster cluster2 snap rm mirror/image_10@snap 2023-08-26T14:00:01.371 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr: Removing snap: 100% complete...done. 2023-08-26T14:00:01.378 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ remove_image_retry cluster2 mirror image_10 2023-08-26T14:00:01.379 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster2 2023-08-26T14:00:01.379 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T14:00:01.379 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=image_10 2023-08-26T14:00:01.379 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ for s in 0 1 2 4 8 16 32 2023-08-26T14:00:01.379 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ sleep 0 2023-08-26T14:00:01.380 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ remove_image cluster2 mirror image_10 2023-08-26T14:00:01.380 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster2 2023-08-26T14:00:01.381 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T14:00:01.381 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=image_10 2023-08-26T14:00:01.381 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_10 2023-08-26T14:00:01.431 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rbd --cluster=cluster2 rm mirror/image_10 2023-08-26T14:00:01.539 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi124.stderr:2023-08-26T14:00:01.538+0000 7fcd42daf700 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x563b71e90d00 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2023-08-26T14:00:01.540 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi124.stderr:2023-08-26T14:00:01.538+0000 7fcd495bc700 -1 rbd::mirror::ImageReplayer: 0x563b71eec000 [3/83466149-7311-4c4c-898e-a67f6a0dbe7c] operator(): start failed: (85) Interrupted system call should be restarted 2023-08-26T14:00:02.497 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2023-08-26T14:00:02.500 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ return 0 2023-08-26T14:00:02.500 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-08-26T14:00:02.500 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ image=image_11 2023-08-26T14:00:02.501 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ remove_snapshot cluster2 mirror image_11 snap 2023-08-26T14:00:02.501 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster2 2023-08-26T14:00:02.501 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T14:00:02.501 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=image_11 2023-08-26T14:00:02.501 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local snap=snap 2023-08-26T14:00:02.502 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rbd --cluster cluster2 snap rm mirror/image_11@snap 2023-08-26T14:00:03.395 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr: Removing snap: 100% complete...done. 2023-08-26T14:00:03.403 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ remove_image_retry cluster2 mirror image_11 2023-08-26T14:00:03.403 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster2 2023-08-26T14:00:03.404 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T14:00:03.404 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=image_11 2023-08-26T14:00:03.404 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ for s in 0 1 2 4 8 16 32 2023-08-26T14:00:03.404 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ sleep 0 2023-08-26T14:00:03.405 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ remove_image cluster2 mirror image_11 2023-08-26T14:00:03.405 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster2 2023-08-26T14:00:03.405 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T14:00:03.406 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=image_11 2023-08-26T14:00:03.406 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_11 2023-08-26T14:00:03.451 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rbd --cluster=cluster2 rm mirror/image_11 2023-08-26T14:00:03.556 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi124.stderr:2023-08-26T14:00:03.555+0000 7f6c3363a700 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x5621ba4fc4e0 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2023-08-26T14:00:03.556 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi124.stderr:2023-08-26T14:00:03.555+0000 7f6c39646700 -1 rbd::mirror::ImageReplayer: 0x5621b9690780 [3/f979d3c9-8b28-4180-872b-79e338751061] operator(): start failed: (85) Interrupted system call should be restarted 2023-08-26T14:00:04.350 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2023-08-26T14:00:04.353 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ return 0 2023-08-26T14:00:04.354 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-08-26T14:00:04.354 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ image=image_12 2023-08-26T14:00:04.354 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ remove_snapshot cluster2 mirror image_12 snap 2023-08-26T14:00:04.354 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster2 2023-08-26T14:00:04.354 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T14:00:04.355 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=image_12 2023-08-26T14:00:04.355 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local snap=snap 2023-08-26T14:00:04.355 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rbd --cluster cluster2 snap rm mirror/image_12@snap 2023-08-26T14:00:05.391 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr: Removing snap: 100% complete...done. 2023-08-26T14:00:05.406 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ remove_image_retry cluster2 mirror image_12 2023-08-26T14:00:05.406 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster2 2023-08-26T14:00:05.406 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T14:00:05.407 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=image_12 2023-08-26T14:00:05.407 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ for s in 0 1 2 4 8 16 32 2023-08-26T14:00:05.407 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ sleep 0 2023-08-26T14:00:05.408 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ remove_image cluster2 mirror image_12 2023-08-26T14:00:05.408 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster2 2023-08-26T14:00:05.408 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T14:00:05.408 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=image_12 2023-08-26T14:00:05.408 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_12 2023-08-26T14:00:05.455 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rbd --cluster=cluster2 rm mirror/image_12 2023-08-26T14:00:05.562 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi124.stderr:2023-08-26T14:00:05.561+0000 7fcd435b0700 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x563b72242680 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2023-08-26T14:00:05.562 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi124.stderr:2023-08-26T14:00:05.561+0000 7fcd49dbd700 -1 rbd::mirror::ImageReplayer: 0x563b71eecc80 [3/b189e689-d710-4185-9857-d4ed9dbb9df5] operator(): start failed: (85) Interrupted system call should be restarted 2023-08-26T14:00:06.519 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2023-08-26T14:00:06.522 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ return 0 2023-08-26T14:00:06.522 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-08-26T14:00:06.522 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ image=image_13 2023-08-26T14:00:06.523 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ remove_snapshot cluster2 mirror image_13 snap 2023-08-26T14:00:06.523 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster2 2023-08-26T14:00:06.523 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T14:00:06.523 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=image_13 2023-08-26T14:00:06.523 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local snap=snap 2023-08-26T14:00:06.523 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rbd --cluster cluster2 snap rm mirror/image_13@snap 2023-08-26T14:00:07.399 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr: Removing snap: 100% complete...done. 2023-08-26T14:00:07.411 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ remove_image_retry cluster2 mirror image_13 2023-08-26T14:00:07.411 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster2 2023-08-26T14:00:07.412 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T14:00:07.412 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=image_13 2023-08-26T14:00:07.412 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ for s in 0 1 2 4 8 16 32 2023-08-26T14:00:07.412 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ sleep 0 2023-08-26T14:00:07.413 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ remove_image cluster2 mirror image_13 2023-08-26T14:00:07.413 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster2 2023-08-26T14:00:07.413 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T14:00:07.414 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=image_13 2023-08-26T14:00:07.414 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_13 2023-08-26T14:00:07.461 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rbd --cluster=cluster2 rm mirror/image_13 2023-08-26T14:00:07.567 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi124.stderr:2023-08-26T14:00:07.566+0000 7fcd42daf700 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x563b722429c0 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2023-08-26T14:00:07.567 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi124.stderr:2023-08-26T14:00:07.566+0000 7fcd49dbd700 -1 rbd::mirror::ImageReplayer: 0x563b71eec500 [3/3f172be8-5137-4cda-906a-0c674ef9c674] operator(): start failed: (85) Interrupted system call should be restarted 2023-08-26T14:00:08.532 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2023-08-26T14:00:08.536 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ return 0 2023-08-26T14:00:08.536 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-08-26T14:00:08.536 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ image=image_14 2023-08-26T14:00:08.536 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ remove_snapshot cluster2 mirror image_14 snap 2023-08-26T14:00:08.537 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster2 2023-08-26T14:00:08.537 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T14:00:08.537 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=image_14 2023-08-26T14:00:08.537 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local snap=snap 2023-08-26T14:00:08.537 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rbd --cluster cluster2 snap rm mirror/image_14@snap 2023-08-26T14:00:09.225 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr: Removing snap: 100% complete...done. 2023-08-26T14:00:09.235 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ remove_image_retry cluster2 mirror image_14 2023-08-26T14:00:09.235 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster2 2023-08-26T14:00:09.235 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T14:00:09.235 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=image_14 2023-08-26T14:00:09.236 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ for s in 0 1 2 4 8 16 32 2023-08-26T14:00:09.236 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ sleep 0 2023-08-26T14:00:09.237 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ remove_image cluster2 mirror image_14 2023-08-26T14:00:09.237 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster2 2023-08-26T14:00:09.237 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T14:00:09.237 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=image_14 2023-08-26T14:00:09.237 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_14 2023-08-26T14:00:09.295 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rbd --cluster=cluster2 rm mirror/image_14 2023-08-26T14:00:09.431 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi124.stderr:2023-08-26T14:00:09.430+0000 7f6c32e39700 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x5621ba521a00 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2023-08-26T14:00:09.431 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi124.stderr:2023-08-26T14:00:09.430+0000 7f6c39646700 -1 rbd::mirror::ImageReplayer: 0x5621ba6b6280 [3/0fecd808-a8ef-45b7-aa0b-52385a8e2b1e] operator(): start failed: (85) Interrupted system call should be restarted 2023-08-26T14:00:10.241 INFO:tasks.ceph.cluster2.osd.2.smithi124.stderr:2023-08-26T14:00:10.238+0000 7fb790854700 -1 reset not still connected to 0x5598e427f110 2023-08-26T14:00:10.241 INFO:tasks.ceph.cluster2.osd.2.smithi124.stderr:2023-08-26T14:00:10.238+0000 7fb790854700 -1 reset not still connected to 0x5599107b9110 2023-08-26T14:00:10.241 INFO:tasks.ceph.cluster2.osd.2.smithi124.stderr:2023-08-26T14:00:10.238+0000 7fb790854700 -1 reset not still connected to 0x55991495bad0 2023-08-26T14:00:10.351 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2023-08-26T14:00:10.355 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ return 0 2023-08-26T14:00:10.355 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-08-26T14:00:10.355 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ image=image_15 2023-08-26T14:00:10.355 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ remove_snapshot cluster2 mirror image_15 snap 2023-08-26T14:00:10.355 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster2 2023-08-26T14:00:10.356 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T14:00:10.356 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=image_15 2023-08-26T14:00:10.356 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local snap=snap 2023-08-26T14:00:10.356 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rbd --cluster cluster2 snap rm mirror/image_15@snap 2023-08-26T14:00:11.226 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr: Removing snap: 100% complete...done. 2023-08-26T14:00:11.238 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ remove_image_retry cluster2 mirror image_15 2023-08-26T14:00:11.238 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster2 2023-08-26T14:00:11.238 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T14:00:11.239 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=image_15 2023-08-26T14:00:11.239 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ for s in 0 1 2 4 8 16 32 2023-08-26T14:00:11.239 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ sleep 0 2023-08-26T14:00:11.240 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ remove_image cluster2 mirror image_15 2023-08-26T14:00:11.241 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster2 2023-08-26T14:00:11.241 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T14:00:11.241 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=image_15 2023-08-26T14:00:11.241 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_15 2023-08-26T14:00:11.290 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rbd --cluster=cluster2 rm mirror/image_15 2023-08-26T14:00:11.425 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi124.stderr:2023-08-26T14:00:11.424+0000 7fc0c3135700 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x5567e0b60000 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2023-08-26T14:00:11.426 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi124.stderr:2023-08-26T14:00:11.424+0000 7fc0c9942700 -1 rbd::mirror::ImageReplayer: 0x5567e030ca00 [3/57b5e313-8920-4ded-8ea4-ab852f9e42c2] operator(): start failed: (85) Interrupted system call should be restarted 2023-08-26T14:00:12.349 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2023-08-26T14:00:12.352 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ return 0 2023-08-26T14:00:12.352 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-08-26T14:00:12.352 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ image=image_16 2023-08-26T14:00:12.353 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ remove_snapshot cluster2 mirror image_16 snap 2023-08-26T14:00:12.353 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster2 2023-08-26T14:00:12.353 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T14:00:12.353 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=image_16 2023-08-26T14:00:12.354 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local snap=snap 2023-08-26T14:00:12.354 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rbd --cluster cluster2 snap rm mirror/image_16@snap 2023-08-26T14:00:13.232 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr: Removing snap: 100% complete...done. 2023-08-26T14:00:13.242 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ remove_image_retry cluster2 mirror image_16 2023-08-26T14:00:13.243 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster2 2023-08-26T14:00:13.243 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T14:00:13.243 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=image_16 2023-08-26T14:00:13.243 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ for s in 0 1 2 4 8 16 32 2023-08-26T14:00:13.243 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ sleep 0 2023-08-26T14:00:13.244 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ remove_image cluster2 mirror image_16 2023-08-26T14:00:13.245 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster2 2023-08-26T14:00:13.245 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T14:00:13.245 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=image_16 2023-08-26T14:00:13.245 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_16 2023-08-26T14:00:13.295 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rbd --cluster=cluster2 rm mirror/image_16 2023-08-26T14:00:13.452 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi124.stderr:2023-08-26T14:00:13.451+0000 7fcd435b0700 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x563b7230aea0 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2023-08-26T14:00:13.452 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi124.stderr:2023-08-26T14:00:13.451+0000 7fcd495bc700 -1 rbd::mirror::ImageReplayer: 0x563b71eec780 [3/bd80dfca-7cec-4de6-be52-c58244873bd3] operator(): start failed: (85) Interrupted system call should be restarted 2023-08-26T14:00:14.361 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2023-08-26T14:00:14.364 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ return 0 2023-08-26T14:00:14.364 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-08-26T14:00:14.364 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ image=image_17 2023-08-26T14:00:14.365 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ remove_snapshot cluster2 mirror image_17 snap 2023-08-26T14:00:14.365 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster2 2023-08-26T14:00:14.365 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T14:00:14.365 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=image_17 2023-08-26T14:00:14.365 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local snap=snap 2023-08-26T14:00:14.366 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rbd --cluster cluster2 snap rm mirror/image_17@snap 2023-08-26T14:00:15.240 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr: Removing snap: 100% complete...done. 2023-08-26T14:00:15.255 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ remove_image_retry cluster2 mirror image_17 2023-08-26T14:00:15.255 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster2 2023-08-26T14:00:15.256 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T14:00:15.256 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=image_17 2023-08-26T14:00:15.256 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ for s in 0 1 2 4 8 16 32 2023-08-26T14:00:15.256 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ sleep 0 2023-08-26T14:00:15.257 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ remove_image cluster2 mirror image_17 2023-08-26T14:00:15.257 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster2 2023-08-26T14:00:15.257 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T14:00:15.258 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=image_17 2023-08-26T14:00:15.258 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_17 2023-08-26T14:00:15.304 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rbd --cluster=cluster2 rm mirror/image_17 2023-08-26T14:00:15.413 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi124.stderr:2023-08-26T14:00:15.412+0000 7f6c3363a700 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x5621ba0a2340 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2023-08-26T14:00:15.413 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi124.stderr:2023-08-26T14:00:15.412+0000 7f6c39646700 -1 rbd::mirror::ImageReplayer: 0x5621ba6b6500 [3/ec21e764-3783-49f1-b802-01400de8810c] operator(): start failed: (85) Interrupted system call should be restarted 2023-08-26T14:00:16.370 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2023-08-26T14:00:16.373 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ return 0 2023-08-26T14:00:16.373 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-08-26T14:00:16.374 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ image=image_18 2023-08-26T14:00:16.374 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ remove_snapshot cluster2 mirror image_18 snap 2023-08-26T14:00:16.374 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster2 2023-08-26T14:00:16.374 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T14:00:16.374 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=image_18 2023-08-26T14:00:16.375 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local snap=snap 2023-08-26T14:00:16.375 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rbd --cluster cluster2 snap rm mirror/image_18@snap 2023-08-26T14:00:17.256 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr: Removing snap: 100% complete...done. 2023-08-26T14:00:17.271 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ remove_image_retry cluster2 mirror image_18 2023-08-26T14:00:17.272 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster2 2023-08-26T14:00:17.272 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T14:00:17.272 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=image_18 2023-08-26T14:00:17.272 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ for s in 0 1 2 4 8 16 32 2023-08-26T14:00:17.272 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ sleep 0 2023-08-26T14:00:17.273 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ remove_image cluster2 mirror image_18 2023-08-26T14:00:17.273 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster2 2023-08-26T14:00:17.274 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T14:00:17.274 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=image_18 2023-08-26T14:00:17.274 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_18 2023-08-26T14:00:17.320 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rbd --cluster=cluster2 rm mirror/image_18 2023-08-26T14:00:17.459 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi124.stderr:2023-08-26T14:00:17.458+0000 7f6c3363a700 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x5621ba5a5d40 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2023-08-26T14:00:17.459 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi124.stderr:2023-08-26T14:00:17.458+0000 7f6c39646700 -1 rbd::mirror::ImageReplayer: 0x5621ba456780 [3/5293326c-bb2a-4d36-bfa9-7cfd4da48fe1] operator(): start failed: (85) Interrupted system call should be restarted 2023-08-26T14:00:17.618 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi124.stderr:2023-08-26T14:00:17.617+0000 7f6c32e39700 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x5621b9e55040 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2023-08-26T14:00:17.618 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi124.stderr:2023-08-26T14:00:17.617+0000 7f6c39646700 -1 rbd::mirror::ImageReplayer: 0x5621ba456780 [3/5293326c-bb2a-4d36-bfa9-7cfd4da48fe1] operator(): start failed: (85) Interrupted system call should be restarted 2023-08-26T14:00:18.383 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2023-08-26T14:00:18.386 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ return 0 2023-08-26T14:00:18.386 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-08-26T14:00:18.387 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ image=image_19 2023-08-26T14:00:18.387 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ remove_snapshot cluster2 mirror image_19 snap 2023-08-26T14:00:18.387 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster2 2023-08-26T14:00:18.387 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T14:00:18.387 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=image_19 2023-08-26T14:00:18.388 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local snap=snap 2023-08-26T14:00:18.388 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rbd --cluster cluster2 snap rm mirror/image_19@snap 2023-08-26T14:00:19.226 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr: Removing snap: 100% complete...done. 2023-08-26T14:00:19.236 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ remove_image_retry cluster2 mirror image_19 2023-08-26T14:00:19.236 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster2 2023-08-26T14:00:19.237 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T14:00:19.237 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=image_19 2023-08-26T14:00:19.237 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ for s in 0 1 2 4 8 16 32 2023-08-26T14:00:19.237 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ sleep 0 2023-08-26T14:00:19.238 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ remove_image cluster2 mirror image_19 2023-08-26T14:00:19.238 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster2 2023-08-26T14:00:19.238 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T14:00:19.238 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=image_19 2023-08-26T14:00:19.239 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_19 2023-08-26T14:00:19.284 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rbd --cluster=cluster2 rm mirror/image_19 2023-08-26T14:00:19.408 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi124.stderr:2023-08-26T14:00:19.407+0000 7fcd42daf700 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x563b7230b1e0 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2023-08-26T14:00:19.408 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi124.stderr:2023-08-26T14:00:19.407+0000 7fcd495bc700 -1 rbd::mirror::ImageReplayer: 0x563b71eed400 [3/f1b3d4dd-7b2d-44de-a0df-388999259524] operator(): start failed: (85) Interrupted system call should be restarted 2023-08-26T14:00:20.387 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2023-08-26T14:00:20.390 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ return 0 2023-08-26T14:00:20.390 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-08-26T14:00:20.390 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ image=image_20 2023-08-26T14:00:20.390 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ remove_snapshot cluster2 mirror image_20 snap 2023-08-26T14:00:20.391 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster2 2023-08-26T14:00:20.391 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T14:00:20.391 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=image_20 2023-08-26T14:00:20.391 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local snap=snap 2023-08-26T14:00:20.391 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rbd --cluster cluster2 snap rm mirror/image_20@snap 2023-08-26T14:00:21.266 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr: Removing snap: 100% complete...done. 2023-08-26T14:00:21.276 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ remove_image_retry cluster2 mirror image_20 2023-08-26T14:00:21.276 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster2 2023-08-26T14:00:21.277 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T14:00:21.277 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=image_20 2023-08-26T14:00:21.277 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ for s in 0 1 2 4 8 16 32 2023-08-26T14:00:21.277 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ sleep 0 2023-08-26T14:00:21.278 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ remove_image cluster2 mirror image_20 2023-08-26T14:00:21.279 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster2 2023-08-26T14:00:21.279 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T14:00:21.279 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=image_20 2023-08-26T14:00:21.279 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_20 2023-08-26T14:00:21.325 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rbd --cluster=cluster2 rm mirror/image_20 2023-08-26T14:00:21.448 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi124.stderr:2023-08-26T14:00:21.447+0000 7fc244bc9700 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x555d5bd8eea0 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2023-08-26T14:00:21.448 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi124.stderr:2023-08-26T14:00:21.447+0000 7fc24bbd7700 -1 rbd::mirror::ImageReplayer: 0x555d5ae9ac80 [3/e8a98866-d970-4e86-bb3b-2de28303ceff] operator(): start failed: (85) Interrupted system call should be restarted 2023-08-26T14:00:22.407 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2023-08-26T14:00:22.410 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ return 0 2023-08-26T14:00:22.410 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-08-26T14:00:22.411 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ image=image_21 2023-08-26T14:00:22.411 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ remove_snapshot cluster2 mirror image_21 snap 2023-08-26T14:00:22.411 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster2 2023-08-26T14:00:22.411 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T14:00:22.411 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=image_21 2023-08-26T14:00:22.412 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local snap=snap 2023-08-26T14:00:22.412 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rbd --cluster cluster2 snap rm mirror/image_21@snap 2023-08-26T14:00:23.316 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr: Removing snap: 100% complete...done. 2023-08-26T14:00:23.326 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ remove_image_retry cluster2 mirror image_21 2023-08-26T14:00:23.326 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster2 2023-08-26T14:00:23.326 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T14:00:23.326 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=image_21 2023-08-26T14:00:23.327 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ for s in 0 1 2 4 8 16 32 2023-08-26T14:00:23.327 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ sleep 0 2023-08-26T14:00:23.327 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ remove_image cluster2 mirror image_21 2023-08-26T14:00:23.328 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster2 2023-08-26T14:00:23.328 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T14:00:23.328 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=image_21 2023-08-26T14:00:23.328 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_21 2023-08-26T14:00:23.386 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rbd --cluster=cluster2 rm mirror/image_21 2023-08-26T14:00:23.531 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi124.stderr:2023-08-26T14:00:23.530+0000 7fc0c3936700 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x5567e0b61a00 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2023-08-26T14:00:23.531 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi124.stderr:2023-08-26T14:00:23.530+0000 7fc0c9942700 -1 rbd::mirror::ImageReplayer: 0x5567e0ba2280 [3/04903506-ddd2-48b7-835f-3797d11e35d3] operator(): start failed: (85) Interrupted system call should be restarted 2023-08-26T14:00:24.359 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2023-08-26T14:00:24.363 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ return 0 2023-08-26T14:00:24.363 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-08-26T14:00:24.363 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ image=image_22 2023-08-26T14:00:24.363 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ remove_snapshot cluster2 mirror image_22 snap 2023-08-26T14:00:24.363 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster2 2023-08-26T14:00:24.364 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T14:00:24.364 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=image_22 2023-08-26T14:00:24.364 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local snap=snap 2023-08-26T14:00:24.364 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rbd --cluster cluster2 snap rm mirror/image_22@snap 2023-08-26T14:00:25.332 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr: Removing snap: 100% complete...done. 2023-08-26T14:00:25.340 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ remove_image_retry cluster2 mirror image_22 2023-08-26T14:00:25.340 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster2 2023-08-26T14:00:25.341 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T14:00:25.341 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=image_22 2023-08-26T14:00:25.341 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ for s in 0 1 2 4 8 16 32 2023-08-26T14:00:25.341 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ sleep 0 2023-08-26T14:00:25.342 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ remove_image cluster2 mirror image_22 2023-08-26T14:00:25.342 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster2 2023-08-26T14:00:25.342 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T14:00:25.343 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=image_22 2023-08-26T14:00:25.343 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_22 2023-08-26T14:00:25.388 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rbd --cluster=cluster2 rm mirror/image_22 2023-08-26T14:00:25.537 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi124.stderr:2023-08-26T14:00:25.536+0000 7f6c3363a700 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x5621b9c5fd40 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2023-08-26T14:00:25.538 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi124.stderr:2023-08-26T14:00:25.536+0000 7f6c39646700 -1 rbd::mirror::ImageReplayer: 0x5621ba456000 [3/70414d4b-2367-4c05-8614-971399383919] operator(): start failed: (85) Interrupted system call should be restarted 2023-08-26T14:00:26.458 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2023-08-26T14:00:26.462 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ return 0 2023-08-26T14:00:26.462 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-08-26T14:00:26.462 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ image=image_23 2023-08-26T14:00:26.463 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ remove_snapshot cluster2 mirror image_23 snap 2023-08-26T14:00:26.463 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster2 2023-08-26T14:00:26.463 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T14:00:26.463 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=image_23 2023-08-26T14:00:26.463 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local snap=snap 2023-08-26T14:00:26.464 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rbd --cluster cluster2 snap rm mirror/image_23@snap 2023-08-26T14:00:27.331 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr: Removing snap: 100% complete...done. 2023-08-26T14:00:27.342 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ remove_image_retry cluster2 mirror image_23 2023-08-26T14:00:27.342 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster2 2023-08-26T14:00:27.342 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T14:00:27.342 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=image_23 2023-08-26T14:00:27.343 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ for s in 0 1 2 4 8 16 32 2023-08-26T14:00:27.343 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ sleep 0 2023-08-26T14:00:27.344 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ remove_image cluster2 mirror image_23 2023-08-26T14:00:27.344 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster2 2023-08-26T14:00:27.344 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T14:00:27.344 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=image_23 2023-08-26T14:00:27.345 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_23 2023-08-26T14:00:27.388 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rbd --cluster=cluster2 rm mirror/image_23 2023-08-26T14:00:27.548 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi124.stderr:2023-08-26T14:00:27.546+0000 7fc2453ca700 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x555d5bd8f1e0 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2023-08-26T14:00:27.548 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi124.stderr:2023-08-26T14:00:27.546+0000 7fc24bbd7700 -1 rbd::mirror::ImageReplayer: 0x555d5ae9aa00 [3/ba978cfa-fbe5-4a3a-86fa-f20310f2d99f] operator(): start failed: (85) Interrupted system call should be restarted 2023-08-26T14:00:28.454 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2023-08-26T14:00:28.457 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ return 0 2023-08-26T14:00:28.458 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-08-26T14:00:28.458 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ image=image_24 2023-08-26T14:00:28.458 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ remove_snapshot cluster2 mirror image_24 snap 2023-08-26T14:00:28.459 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster2 2023-08-26T14:00:28.459 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T14:00:28.459 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=image_24 2023-08-26T14:00:28.459 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local snap=snap 2023-08-26T14:00:28.460 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rbd --cluster cluster2 snap rm mirror/image_24@snap 2023-08-26T14:00:28.549 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.0 with skip (0.8401707200635256) > weight (0.25) 2023-08-26T14:00:28.550 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.1 with skip (0.7313588694738447) > weight (0.25) 2023-08-26T14:00:28.550 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.2 with skip (0.6377418941327208) > weight (0.25) 2023-08-26T14:00:28.550 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.3 with skip (0.449589640723041) > weight (0.25) 2023-08-26T14:00:28.550 INFO:tasks.rbd_mirror_thrash:waiting for 83 secs before thrashing 2023-08-26T14:00:29.235 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr: Removing snap: 100% complete...done. 2023-08-26T14:00:29.245 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ remove_image_retry cluster2 mirror image_24 2023-08-26T14:00:29.245 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster2 2023-08-26T14:00:29.246 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T14:00:29.246 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=image_24 2023-08-26T14:00:29.246 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ for s in 0 1 2 4 8 16 32 2023-08-26T14:00:29.246 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ sleep 0 2023-08-26T14:00:29.247 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ remove_image cluster2 mirror image_24 2023-08-26T14:00:29.247 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster2 2023-08-26T14:00:29.248 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T14:00:29.248 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=image_24 2023-08-26T14:00:29.248 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_24 2023-08-26T14:00:29.296 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rbd --cluster=cluster2 rm mirror/image_24 2023-08-26T14:00:29.404 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi124.stderr:2023-08-26T14:00:29.402+0000 7fcd42daf700 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x563b7230b520 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2023-08-26T14:00:29.405 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi124.stderr:2023-08-26T14:00:29.402+0000 7fcd495bc700 -1 rbd::mirror::ImageReplayer: 0x563b71ce4280 [3/bfc1dd79-696c-4322-bf38-f1dc8a31a580] operator(): start failed: (85) Interrupted system call should be restarted 2023-08-26T14:00:30.469 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2023-08-26T14:00:30.472 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ return 0 2023-08-26T14:00:30.472 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-08-26T14:00:30.473 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ image=image_25 2023-08-26T14:00:30.473 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ remove_snapshot cluster2 mirror image_25 snap 2023-08-26T14:00:30.473 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster2 2023-08-26T14:00:30.473 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T14:00:30.473 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=image_25 2023-08-26T14:00:30.473 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local snap=snap 2023-08-26T14:00:30.474 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rbd --cluster cluster2 snap rm mirror/image_25@snap 2023-08-26T14:00:31.342 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr: Removing snap: 100% complete...done. 2023-08-26T14:00:31.354 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ remove_image_retry cluster2 mirror image_25 2023-08-26T14:00:31.354 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster2 2023-08-26T14:00:31.354 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T14:00:31.354 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=image_25 2023-08-26T14:00:31.355 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ for s in 0 1 2 4 8 16 32 2023-08-26T14:00:31.355 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ sleep 0 2023-08-26T14:00:31.356 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ remove_image cluster2 mirror image_25 2023-08-26T14:00:31.356 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster2 2023-08-26T14:00:31.356 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T14:00:31.356 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=image_25 2023-08-26T14:00:31.357 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_25 2023-08-26T14:00:31.402 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rbd --cluster=cluster2 rm mirror/image_25 2023-08-26T14:00:31.520 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi124.stderr:2023-08-26T14:00:31.518+0000 7fc0c3936700 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x5567e0d29ba0 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2023-08-26T14:00:31.521 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi124.stderr:2023-08-26T14:00:31.518+0000 7fc0ca143700 -1 rbd::mirror::ImageReplayer: 0x5567e030d400 [3/ae7a1e38-d406-47df-97d4-9cbf52592bb7] operator(): start failed: (85) Interrupted system call should be restarted 2023-08-26T14:00:32.485 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2023-08-26T14:00:32.489 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ return 0 2023-08-26T14:00:32.489 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-08-26T14:00:32.490 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ image=image_26 2023-08-26T14:00:32.490 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ remove_snapshot cluster2 mirror image_26 snap 2023-08-26T14:00:32.490 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster2 2023-08-26T14:00:32.491 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T14:00:32.491 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=image_26 2023-08-26T14:00:32.491 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local snap=snap 2023-08-26T14:00:32.492 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rbd --cluster cluster2 snap rm mirror/image_26@snap 2023-08-26T14:00:33.357 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr: Removing snap: 100% complete...done. 2023-08-26T14:00:33.367 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ remove_image_retry cluster2 mirror image_26 2023-08-26T14:00:33.368 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster2 2023-08-26T14:00:33.368 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T14:00:33.368 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=image_26 2023-08-26T14:00:33.368 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ for s in 0 1 2 4 8 16 32 2023-08-26T14:00:33.368 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ sleep 0 2023-08-26T14:00:33.369 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ remove_image cluster2 mirror image_26 2023-08-26T14:00:33.369 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster2 2023-08-26T14:00:33.370 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T14:00:33.370 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=image_26 2023-08-26T14:00:33.370 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_26 2023-08-26T14:00:33.417 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rbd --cluster=cluster2 rm mirror/image_26 2023-08-26T14:00:33.522 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi124.stderr:2023-08-26T14:00:33.520+0000 7fc0c3936700 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x5567e0b60820 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2023-08-26T14:00:33.522 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi124.stderr:2023-08-26T14:00:33.520+0000 7fc0c9942700 -1 rbd::mirror::ImageReplayer: 0x5567e0ba2000 [3/ef7edecb-ca31-4518-87d6-58c3dfe520f7] operator(): start failed: (85) Interrupted system call should be restarted 2023-08-26T14:00:34.360 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2023-08-26T14:00:34.364 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ return 0 2023-08-26T14:00:34.364 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-08-26T14:00:34.364 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ image=image_27 2023-08-26T14:00:34.364 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ remove_snapshot cluster2 mirror image_27 snap 2023-08-26T14:00:34.365 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster2 2023-08-26T14:00:34.365 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T14:00:34.365 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=image_27 2023-08-26T14:00:34.365 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local snap=snap 2023-08-26T14:00:34.365 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rbd --cluster cluster2 snap rm mirror/image_27@snap 2023-08-26T14:00:35.364 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr: Removing snap: 100% complete...done. 2023-08-26T14:00:35.375 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ remove_image_retry cluster2 mirror image_27 2023-08-26T14:00:35.375 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster2 2023-08-26T14:00:35.375 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T14:00:35.375 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=image_27 2023-08-26T14:00:35.376 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ for s in 0 1 2 4 8 16 32 2023-08-26T14:00:35.376 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ sleep 0 2023-08-26T14:00:35.377 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ remove_image cluster2 mirror image_27 2023-08-26T14:00:35.377 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster2 2023-08-26T14:00:35.377 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T14:00:35.377 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=image_27 2023-08-26T14:00:35.377 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_27 2023-08-26T14:00:35.432 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rbd --cluster=cluster2 rm mirror/image_27 2023-08-26T14:00:35.537 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi124.stderr:2023-08-26T14:00:35.535+0000 7f6c32e39700 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x5621b9e54340 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2023-08-26T14:00:35.538 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi124.stderr:2023-08-26T14:00:35.535+0000 7f6c39646700 -1 rbd::mirror::ImageReplayer: 0x5621ba456c80 [3/bceda8a0-e824-4b77-b33c-1ee70e5c9a49] operator(): start failed: (85) Interrupted system call should be restarted 2023-08-26T14:00:36.498 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2023-08-26T14:00:36.502 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ return 0 2023-08-26T14:00:36.503 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-08-26T14:00:36.503 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ image=image_28 2023-08-26T14:00:36.503 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ remove_snapshot cluster2 mirror image_28 snap 2023-08-26T14:00:36.503 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster2 2023-08-26T14:00:36.504 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T14:00:36.504 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=image_28 2023-08-26T14:00:36.504 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local snap=snap 2023-08-26T14:00:36.504 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rbd --cluster cluster2 snap rm mirror/image_28@snap 2023-08-26T14:00:37.359 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr: Removing snap: 100% complete...done. 2023-08-26T14:00:37.372 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ remove_image_retry cluster2 mirror image_28 2023-08-26T14:00:37.372 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster2 2023-08-26T14:00:37.372 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T14:00:37.373 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=image_28 2023-08-26T14:00:37.373 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ for s in 0 1 2 4 8 16 32 2023-08-26T14:00:37.373 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ sleep 0 2023-08-26T14:00:37.374 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ remove_image cluster2 mirror image_28 2023-08-26T14:00:37.374 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster2 2023-08-26T14:00:37.374 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T14:00:37.374 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=image_28 2023-08-26T14:00:37.375 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_28 2023-08-26T14:00:37.433 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rbd --cluster=cluster2 rm mirror/image_28 2023-08-26T14:00:37.543 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi124.stderr:2023-08-26T14:00:37.540+0000 7fc244bc9700 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x555d5bf56680 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2023-08-26T14:00:37.543 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi124.stderr:2023-08-26T14:00:37.540+0000 7fc24b3d6700 -1 rbd::mirror::ImageReplayer: 0x555d5ae9b900 [3/b66f96d3-c7f1-4362-870d-f32a84410a1f] operator(): start failed: (85) Interrupted system call should be restarted 2023-08-26T14:00:38.510 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2023-08-26T14:00:38.515 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ return 0 2023-08-26T14:00:38.515 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-08-26T14:00:38.515 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ image=image_29 2023-08-26T14:00:38.516 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ remove_snapshot cluster2 mirror image_29 snap 2023-08-26T14:00:38.516 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster2 2023-08-26T14:00:38.516 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T14:00:38.516 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=image_29 2023-08-26T14:00:38.516 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local snap=snap 2023-08-26T14:00:38.517 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rbd --cluster cluster2 snap rm mirror/image_29@snap 2023-08-26T14:00:39.222 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr: Removing snap: 100% complete...done. 2023-08-26T14:00:39.235 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ remove_image_retry cluster2 mirror image_29 2023-08-26T14:00:39.235 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster2 2023-08-26T14:00:39.235 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T14:00:39.236 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=image_29 2023-08-26T14:00:39.236 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ for s in 0 1 2 4 8 16 32 2023-08-26T14:00:39.236 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ sleep 0 2023-08-26T14:00:39.237 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ remove_image cluster2 mirror image_29 2023-08-26T14:00:39.237 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster2 2023-08-26T14:00:39.237 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T14:00:39.238 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=image_29 2023-08-26T14:00:39.238 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_29 2023-08-26T14:00:39.282 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rbd --cluster=cluster2 rm mirror/image_29 2023-08-26T14:00:39.438 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi124.stderr:2023-08-26T14:00:39.436+0000 7fc244bc9700 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x555d5bf56b60 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2023-08-26T14:00:39.439 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi124.stderr:2023-08-26T14:00:39.436+0000 7fc24b3d6700 -1 rbd::mirror::ImageReplayer: 0x555d5ae9af00 [3/2286d3a2-d0dc-442e-9a23-9c83c4633272] operator(): start failed: (85) Interrupted system call should be restarted 2023-08-26T14:00:40.515 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2023-08-26T14:00:40.518 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ return 0 2023-08-26T14:00:40.518 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-08-26T14:00:40.519 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ image=image_30 2023-08-26T14:00:40.519 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ remove_snapshot cluster2 mirror image_30 snap 2023-08-26T14:00:40.519 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster2 2023-08-26T14:00:40.519 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T14:00:40.519 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=image_30 2023-08-26T14:00:40.520 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local snap=snap 2023-08-26T14:00:40.520 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rbd --cluster cluster2 snap rm mirror/image_30@snap 2023-08-26T14:00:41.379 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr: Removing snap: 100% complete...done. 2023-08-26T14:00:41.389 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ remove_image_retry cluster2 mirror image_30 2023-08-26T14:00:41.390 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster2 2023-08-26T14:00:41.390 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T14:00:41.390 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=image_30 2023-08-26T14:00:41.390 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ for s in 0 1 2 4 8 16 32 2023-08-26T14:00:41.390 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ sleep 0 2023-08-26T14:00:41.391 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ remove_image cluster2 mirror image_30 2023-08-26T14:00:41.391 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster2 2023-08-26T14:00:41.392 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T14:00:41.392 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=image_30 2023-08-26T14:00:41.392 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_30 2023-08-26T14:00:41.439 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rbd --cluster=cluster2 rm mirror/image_30 2023-08-26T14:00:41.561 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi124.stderr:2023-08-26T14:00:41.559+0000 7fcd435b0700 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x563b72260000 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2023-08-26T14:00:41.562 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi124.stderr:2023-08-26T14:00:41.559+0000 7fcd49dbd700 -1 rbd::mirror::ImageReplayer: 0x563b71ce4780 [3/530f9c41-4e08-4e1d-99cd-a782ba07a616] operator(): start failed: (85) Interrupted system call should be restarted 2023-08-26T14:00:42.516 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2023-08-26T14:00:42.519 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ return 0 2023-08-26T14:00:42.520 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-08-26T14:00:42.520 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ image=image_31 2023-08-26T14:00:42.520 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ remove_snapshot cluster2 mirror image_31 snap 2023-08-26T14:00:42.520 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster2 2023-08-26T14:00:42.520 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T14:00:42.521 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=image_31 2023-08-26T14:00:42.521 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local snap=snap 2023-08-26T14:00:42.521 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rbd --cluster cluster2 snap rm mirror/image_31@snap 2023-08-26T14:00:43.379 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr: Removing snap: 100% complete...done. 2023-08-26T14:00:43.394 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ remove_image_retry cluster2 mirror image_31 2023-08-26T14:00:43.394 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster2 2023-08-26T14:00:43.394 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T14:00:43.395 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=image_31 2023-08-26T14:00:43.395 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ for s in 0 1 2 4 8 16 32 2023-08-26T14:00:43.395 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ sleep 0 2023-08-26T14:00:43.396 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ remove_image cluster2 mirror image_31 2023-08-26T14:00:43.396 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster2 2023-08-26T14:00:43.396 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T14:00:43.396 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=image_31 2023-08-26T14:00:43.397 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_31 2023-08-26T14:00:43.452 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rbd --cluster=cluster2 rm mirror/image_31 2023-08-26T14:00:43.557 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi124.stderr:2023-08-26T14:00:43.554+0000 7f6c32e39700 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x5621b9e7ad00 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2023-08-26T14:00:43.557 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi124.stderr:2023-08-26T14:00:43.554+0000 7f6c39e47700 -1 rbd::mirror::ImageReplayer: 0x5621ba457900 [3/c5831e57-7926-4f65-bb0c-ef45cd41b0d4] operator(): start failed: (85) Interrupted system call should be restarted 2023-08-26T14:00:44.355 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2023-08-26T14:00:44.358 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ return 0 2023-08-26T14:00:44.358 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-08-26T14:00:44.358 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ image=image_32 2023-08-26T14:00:44.359 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ remove_snapshot cluster2 mirror image_32 snap 2023-08-26T14:00:44.359 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster2 2023-08-26T14:00:44.359 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T14:00:44.360 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=image_32 2023-08-26T14:00:44.360 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local snap=snap 2023-08-26T14:00:44.360 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rbd --cluster cluster2 snap rm mirror/image_32@snap 2023-08-26T14:00:45.393 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr: Removing snap: 100% complete...done. 2023-08-26T14:00:45.405 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ remove_image_retry cluster2 mirror image_32 2023-08-26T14:00:45.405 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster2 2023-08-26T14:00:45.405 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T14:00:45.405 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=image_32 2023-08-26T14:00:45.405 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ for s in 0 1 2 4 8 16 32 2023-08-26T14:00:45.406 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ sleep 0 2023-08-26T14:00:45.406 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ remove_image cluster2 mirror image_32 2023-08-26T14:00:45.406 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster2 2023-08-26T14:00:45.407 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T14:00:45.407 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=image_32 2023-08-26T14:00:45.407 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_32 2023-08-26T14:00:45.454 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rbd --cluster=cluster2 rm mirror/image_32 2023-08-26T14:00:45.578 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi124.stderr:2023-08-26T14:00:45.573+0000 7f6c32e39700 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x5621ba1596c0 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2023-08-26T14:00:45.578 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi124.stderr:2023-08-26T14:00:45.573+0000 7f6c39e47700 -1 rbd::mirror::ImageReplayer: 0x5621b9690f00 [3/9ec01316-ea12-44fa-bbc4-3dc34cb1f619] operator(): start failed: (85) Interrupted system call should be restarted 2023-08-26T14:00:46.546 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2023-08-26T14:00:46.550 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ return 0 2023-08-26T14:00:46.550 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-08-26T14:00:46.550 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ image=image_33 2023-08-26T14:00:46.551 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ remove_snapshot cluster2 mirror image_33 snap 2023-08-26T14:00:46.551 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster2 2023-08-26T14:00:46.551 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T14:00:46.551 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=image_33 2023-08-26T14:00:46.552 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local snap=snap 2023-08-26T14:00:46.552 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rbd --cluster cluster2 snap rm mirror/image_33@snap 2023-08-26T14:00:47.408 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr: Removing snap: 100% complete...done. 2023-08-26T14:00:47.422 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ remove_image_retry cluster2 mirror image_33 2023-08-26T14:00:47.422 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster2 2023-08-26T14:00:47.422 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T14:00:47.423 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=image_33 2023-08-26T14:00:47.423 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ for s in 0 1 2 4 8 16 32 2023-08-26T14:00:47.423 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ sleep 0 2023-08-26T14:00:47.424 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ remove_image cluster2 mirror image_33 2023-08-26T14:00:47.424 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster2 2023-08-26T14:00:47.425 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T14:00:47.425 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=image_33 2023-08-26T14:00:47.425 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_33 2023-08-26T14:00:47.476 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rbd --cluster=cluster2 rm mirror/image_33 2023-08-26T14:00:47.583 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi124.stderr:2023-08-26T14:00:47.581+0000 7fc244bc9700 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x555d5bd8e1a0 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2023-08-26T14:00:47.584 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi124.stderr:2023-08-26T14:00:47.581+0000 7fc24bbd7700 -1 rbd::mirror::ImageReplayer: 0x555d5b991180 [3/3c30bb33-dd5f-4030-adb6-12b16331cd53] operator(): start failed: (85) Interrupted system call should be restarted 2023-08-26T14:00:48.546 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2023-08-26T14:00:48.550 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ return 0 2023-08-26T14:00:48.550 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-08-26T14:00:48.550 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ image=image_34 2023-08-26T14:00:48.551 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ remove_snapshot cluster2 mirror image_34 snap 2023-08-26T14:00:48.551 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster2 2023-08-26T14:00:48.551 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T14:00:48.551 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=image_34 2023-08-26T14:00:48.551 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local snap=snap 2023-08-26T14:00:48.552 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rbd --cluster cluster2 snap rm mirror/image_34@snap 2023-08-26T14:00:49.227 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr: Removing snap: 100% complete...done. 2023-08-26T14:00:49.236 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ remove_image_retry cluster2 mirror image_34 2023-08-26T14:00:49.237 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster2 2023-08-26T14:00:49.237 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T14:00:49.237 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=image_34 2023-08-26T14:00:49.238 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ for s in 0 1 2 4 8 16 32 2023-08-26T14:00:49.238 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ sleep 0 2023-08-26T14:00:49.239 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ remove_image cluster2 mirror image_34 2023-08-26T14:00:49.239 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster2 2023-08-26T14:00:49.239 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T14:00:49.240 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=image_34 2023-08-26T14:00:49.240 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_34 2023-08-26T14:00:49.282 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rbd --cluster=cluster2 rm mirror/image_34 2023-08-26T14:00:49.436 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi124.stderr:2023-08-26T14:00:49.434+0000 7fc244bc9700 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x555d5bd8f380 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2023-08-26T14:00:49.436 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi124.stderr:2023-08-26T14:00:49.434+0000 7fc24bbd7700 -1 rbd::mirror::ImageReplayer: 0x555d5ae9a500 [3/ee57f6e1-9b01-446a-9317-2f9969ecc132] operator(): start failed: (85) Interrupted system call should be restarted 2023-08-26T14:00:50.365 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2023-08-26T14:00:50.369 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ return 0 2023-08-26T14:00:50.369 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-08-26T14:00:50.369 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ image=image_35 2023-08-26T14:00:50.369 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ remove_snapshot cluster2 mirror image_35 snap 2023-08-26T14:00:50.370 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster2 2023-08-26T14:00:50.370 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T14:00:50.370 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=image_35 2023-08-26T14:00:50.370 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local snap=snap 2023-08-26T14:00:50.370 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rbd --cluster cluster2 snap rm mirror/image_35@snap 2023-08-26T14:00:51.233 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr: Removing snap: 100% complete...done. 2023-08-26T14:00:51.244 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ remove_image_retry cluster2 mirror image_35 2023-08-26T14:00:51.244 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster2 2023-08-26T14:00:51.244 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T14:00:51.244 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=image_35 2023-08-26T14:00:51.245 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ for s in 0 1 2 4 8 16 32 2023-08-26T14:00:51.245 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ sleep 0 2023-08-26T14:00:51.246 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ remove_image cluster2 mirror image_35 2023-08-26T14:00:51.246 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster2 2023-08-26T14:00:51.246 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T14:00:51.246 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=image_35 2023-08-26T14:00:51.246 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_35 2023-08-26T14:00:51.294 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rbd --cluster=cluster2 rm mirror/image_35 2023-08-26T14:00:51.439 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi124.stderr:2023-08-26T14:00:51.437+0000 7fc0c3936700 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x5567e0dda4e0 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2023-08-26T14:00:51.439 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi124.stderr:2023-08-26T14:00:51.437+0000 7fc0ca143700 -1 rbd::mirror::ImageReplayer: 0x5567e030cf00 [3/92367d6b-5947-4538-a25d-c7bdd622153a] operator(): start failed: (85) Interrupted system call should be restarted 2023-08-26T14:00:52.381 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2023-08-26T14:00:52.384 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ return 0 2023-08-26T14:00:52.385 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-08-26T14:00:52.385 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ image=image_36 2023-08-26T14:00:52.385 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ remove_snapshot cluster2 mirror image_36 snap 2023-08-26T14:00:52.385 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster2 2023-08-26T14:00:52.385 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T14:00:52.386 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=image_36 2023-08-26T14:00:52.386 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local snap=snap 2023-08-26T14:00:52.386 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rbd --cluster cluster2 snap rm mirror/image_36@snap 2023-08-26T14:00:53.250 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr: Removing snap: 100% complete...done. 2023-08-26T14:00:53.266 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ remove_image_retry cluster2 mirror image_36 2023-08-26T14:00:53.266 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster2 2023-08-26T14:00:53.267 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T14:00:53.267 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=image_36 2023-08-26T14:00:53.267 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ for s in 0 1 2 4 8 16 32 2023-08-26T14:00:53.267 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ sleep 0 2023-08-26T14:00:53.268 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ remove_image cluster2 mirror image_36 2023-08-26T14:00:53.268 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster2 2023-08-26T14:00:53.268 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T14:00:53.269 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=image_36 2023-08-26T14:00:53.269 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_36 2023-08-26T14:00:53.315 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rbd --cluster=cluster2 rm mirror/image_36 2023-08-26T14:00:53.481 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi124.stderr:2023-08-26T14:00:53.479+0000 7fc0c3936700 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x5567e0d48340 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2023-08-26T14:00:53.482 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi124.stderr:2023-08-26T14:00:53.479+0000 7fc0c9942700 -1 rbd::mirror::ImageReplayer: 0x5567e030d180 [3/8068d21b-b929-4466-9d32-bbd96d0ca61c] operator(): start failed: (85) Interrupted system call should be restarted 2023-08-26T14:00:54.364 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2023-08-26T14:00:54.367 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ return 0 2023-08-26T14:00:54.367 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-08-26T14:00:54.368 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ image=image_37 2023-08-26T14:00:54.368 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ remove_snapshot cluster2 mirror image_37 snap 2023-08-26T14:00:54.368 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster2 2023-08-26T14:00:54.368 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T14:00:54.368 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=image_37 2023-08-26T14:00:54.368 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local snap=snap 2023-08-26T14:00:54.369 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rbd --cluster cluster2 snap rm mirror/image_37@snap 2023-08-26T14:00:55.259 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr: Removing snap: 100% complete...done. 2023-08-26T14:00:55.270 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ remove_image_retry cluster2 mirror image_37 2023-08-26T14:00:55.271 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster2 2023-08-26T14:00:55.271 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T14:00:55.271 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=image_37 2023-08-26T14:00:55.271 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ for s in 0 1 2 4 8 16 32 2023-08-26T14:00:55.272 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ sleep 0 2023-08-26T14:00:55.272 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ remove_image cluster2 mirror image_37 2023-08-26T14:00:55.273 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster2 2023-08-26T14:00:55.273 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T14:00:55.273 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=image_37 2023-08-26T14:00:55.273 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_37 2023-08-26T14:00:55.318 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rbd --cluster=cluster2 rm mirror/image_37 2023-08-26T14:00:55.396 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi124.stderr:2023-08-26T14:00:55.395+0000 7f5fa578d700 -1 rbd::mirror::ImageReplayer: 0x563345063680 [3/339b230b-a92d-47ff-8812-3e7a71f8b968] start: already running 2023-08-26T14:00:55.437 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi124.stderr:2023-08-26T14:00:55.436+0000 7fc2453ca700 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x555d5bf564e0 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2023-08-26T14:00:55.438 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi124.stderr:2023-08-26T14:00:55.436+0000 7fc24b3d6700 -1 rbd::mirror::ImageReplayer: 0x555d5ae9a280 [3/339b230b-a92d-47ff-8812-3e7a71f8b968] operator(): start failed: (85) Interrupted system call should be restarted 2023-08-26T14:00:56.377 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2023-08-26T14:00:56.381 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ return 0 2023-08-26T14:00:56.381 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-08-26T14:00:56.381 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ image=image_38 2023-08-26T14:00:56.381 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ remove_snapshot cluster2 mirror image_38 snap 2023-08-26T14:00:56.381 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster2 2023-08-26T14:00:56.382 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T14:00:56.382 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=image_38 2023-08-26T14:00:56.382 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local snap=snap 2023-08-26T14:00:56.382 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rbd --cluster cluster2 snap rm mirror/image_38@snap 2023-08-26T14:00:57.267 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr: Removing snap: 100% complete...done. 2023-08-26T14:00:57.276 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ remove_image_retry cluster2 mirror image_38 2023-08-26T14:00:57.276 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster2 2023-08-26T14:00:57.276 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T14:00:57.277 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=image_38 2023-08-26T14:00:57.277 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ for s in 0 1 2 4 8 16 32 2023-08-26T14:00:57.277 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ sleep 0 2023-08-26T14:00:57.278 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ remove_image cluster2 mirror image_38 2023-08-26T14:00:57.278 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster2 2023-08-26T14:00:57.279 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T14:00:57.279 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=image_38 2023-08-26T14:00:57.279 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_38 2023-08-26T14:00:57.336 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rbd --cluster=cluster2 rm mirror/image_38 2023-08-26T14:00:57.486 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi124.stderr:2023-08-26T14:00:57.485+0000 7f6c3363a700 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x5621ba7dbd40 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2023-08-26T14:00:57.487 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi124.stderr:2023-08-26T14:00:57.485+0000 7f6c39e47700 -1 rbd::mirror::ImageReplayer: 0x5621ba456f00 [3/1e20d016-f50e-43c9-b2e8-47e995fee9bd] operator(): start failed: (85) Interrupted system call should be restarted 2023-08-26T14:00:58.395 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2023-08-26T14:00:58.399 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ return 0 2023-08-26T14:00:58.399 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-08-26T14:00:58.399 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ image=image_39 2023-08-26T14:00:58.400 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ remove_snapshot cluster2 mirror image_39 snap 2023-08-26T14:00:58.400 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster2 2023-08-26T14:00:58.400 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T14:00:58.400 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=image_39 2023-08-26T14:00:58.400 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local snap=snap 2023-08-26T14:00:58.401 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rbd --cluster cluster2 snap rm mirror/image_39@snap 2023-08-26T14:00:59.232 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr: Removing snap: 100% complete...done. 2023-08-26T14:00:59.247 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ remove_image_retry cluster2 mirror image_39 2023-08-26T14:00:59.247 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster2 2023-08-26T14:00:59.247 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T14:00:59.247 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=image_39 2023-08-26T14:00:59.247 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ for s in 0 1 2 4 8 16 32 2023-08-26T14:00:59.248 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ sleep 0 2023-08-26T14:00:59.249 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ remove_image cluster2 mirror image_39 2023-08-26T14:00:59.249 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster2 2023-08-26T14:00:59.249 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T14:00:59.249 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=image_39 2023-08-26T14:00:59.249 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_39 2023-08-26T14:00:59.293 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rbd --cluster=cluster2 rm mirror/image_39 2023-08-26T14:00:59.410 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi124.stderr:2023-08-26T14:00:59.409+0000 7fc0c3135700 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x5567e0d48000 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2023-08-26T14:00:59.410 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi124.stderr:2023-08-26T14:00:59.409+0000 7fc0c9942700 -1 rbd::mirror::ImageReplayer: 0x5567e030d680 [3/e777bb16-afba-4c52-994a-6980c13f1760] operator(): start failed: (85) Interrupted system call should be restarted 2023-08-26T14:01:00.420 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2023-08-26T14:01:00.423 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ return 0 2023-08-26T14:01:00.423 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-08-26T14:01:00.423 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ image=image_40 2023-08-26T14:01:00.424 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ remove_snapshot cluster2 mirror image_40 snap 2023-08-26T14:01:00.424 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster2 2023-08-26T14:01:00.424 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T14:01:00.424 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=image_40 2023-08-26T14:01:00.424 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local snap=snap 2023-08-26T14:01:00.425 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rbd --cluster cluster2 snap rm mirror/image_40@snap 2023-08-26T14:01:01.282 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr: Removing snap: 100% complete...done. 2023-08-26T14:01:01.288 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ remove_image_retry cluster2 mirror image_40 2023-08-26T14:01:01.288 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster2 2023-08-26T14:01:01.288 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T14:01:01.289 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=image_40 2023-08-26T14:01:01.289 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ for s in 0 1 2 4 8 16 32 2023-08-26T14:01:01.289 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ sleep 0 2023-08-26T14:01:01.290 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ remove_image cluster2 mirror image_40 2023-08-26T14:01:01.290 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster2 2023-08-26T14:01:01.290 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T14:01:01.290 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=image_40 2023-08-26T14:01:01.291 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_40 2023-08-26T14:01:01.333 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rbd --cluster=cluster2 rm mirror/image_40 2023-08-26T14:01:01.454 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi124.stderr:2023-08-26T14:01:01.453+0000 7fcd435b0700 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x563b722601a0 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2023-08-26T14:01:01.454 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi124.stderr:2023-08-26T14:01:01.453+0000 7fcd49dbd700 -1 rbd::mirror::ImageReplayer: 0x563b71ce4000 [3/5bb319f8-2f9d-4a84-8756-0c66e011e111] operator(): start failed: (85) Interrupted system call should be restarted 2023-08-26T14:01:01.728 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi124.stderr:2023-08-26T14:01:01.727+0000 7fcd435b0700 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x563b7230ad00 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2023-08-26T14:01:01.729 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi124.stderr:2023-08-26T14:01:01.727+0000 7fcd49dbd700 -1 rbd::mirror::ImageReplayer: 0x563b71ce4000 [3/5bb319f8-2f9d-4a84-8756-0c66e011e111] operator(): start failed: (85) Interrupted system call should be restarted 2023-08-26T14:01:02.427 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2023-08-26T14:01:02.430 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ return 0 2023-08-26T14:01:02.430 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-08-26T14:01:02.431 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ image=image_41 2023-08-26T14:01:02.431 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ remove_snapshot cluster2 mirror image_41 snap 2023-08-26T14:01:02.431 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster2 2023-08-26T14:01:02.431 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T14:01:02.432 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=image_41 2023-08-26T14:01:02.432 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local snap=snap 2023-08-26T14:01:02.432 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rbd --cluster cluster2 snap rm mirror/image_41@snap 2023-08-26T14:01:03.295 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr: Removing snap: 100% complete...done. 2023-08-26T14:01:03.308 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ remove_image_retry cluster2 mirror image_41 2023-08-26T14:01:03.308 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster2 2023-08-26T14:01:03.309 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T14:01:03.309 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=image_41 2023-08-26T14:01:03.309 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ for s in 0 1 2 4 8 16 32 2023-08-26T14:01:03.309 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ sleep 0 2023-08-26T14:01:03.310 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ remove_image cluster2 mirror image_41 2023-08-26T14:01:03.310 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster2 2023-08-26T14:01:03.310 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T14:01:03.310 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=image_41 2023-08-26T14:01:03.311 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_41 2023-08-26T14:01:03.355 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rbd --cluster=cluster2 rm mirror/image_41 2023-08-26T14:01:03.464 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi124.stderr:2023-08-26T14:01:03.463+0000 7fc0c3135700 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x5567e0b604e0 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2023-08-26T14:01:03.465 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi124.stderr:2023-08-26T14:01:03.463+0000 7fc0c9942700 -1 rbd::mirror::ImageReplayer: 0x5567e030d900 [3/bfd70442-dd63-468e-9119-77e3357316c5] operator(): start failed: (85) Interrupted system call should be restarted 2023-08-26T14:01:04.263 INFO:tasks.ceph.cluster2.osd.2.smithi124.stderr:2023-08-26T14:01:04.262+0000 7fb790854700 -1 reset not still connected to 0x5599107b9110 2023-08-26T14:01:04.264 INFO:tasks.ceph.cluster2.osd.2.smithi124.stderr:2023-08-26T14:01:04.262+0000 7fb790854700 -1 reset not still connected to 0x55991757ed00 2023-08-26T14:01:04.365 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2023-08-26T14:01:04.369 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ return 0 2023-08-26T14:01:04.369 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-08-26T14:01:04.369 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ image=image_42 2023-08-26T14:01:04.370 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ remove_snapshot cluster2 mirror image_42 snap 2023-08-26T14:01:04.370 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster2 2023-08-26T14:01:04.370 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T14:01:04.370 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=image_42 2023-08-26T14:01:04.370 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local snap=snap 2023-08-26T14:01:04.371 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rbd --cluster cluster2 snap rm mirror/image_42@snap 2023-08-26T14:01:05.295 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr: Removing snap: 100% complete...done. 2023-08-26T14:01:05.304 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ remove_image_retry cluster2 mirror image_42 2023-08-26T14:01:05.304 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster2 2023-08-26T14:01:05.304 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T14:01:05.304 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=image_42 2023-08-26T14:01:05.305 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ for s in 0 1 2 4 8 16 32 2023-08-26T14:01:05.305 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ sleep 0 2023-08-26T14:01:05.305 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ remove_image cluster2 mirror image_42 2023-08-26T14:01:05.306 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster2 2023-08-26T14:01:05.306 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T14:01:05.306 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=image_42 2023-08-26T14:01:05.306 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_42 2023-08-26T14:01:05.356 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rbd --cluster=cluster2 rm mirror/image_42 2023-08-26T14:01:05.464 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi124.stderr:2023-08-26T14:01:05.463+0000 7f6c3363a700 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x5621ba7da680 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2023-08-26T14:01:06.445 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2023-08-26T14:01:06.449 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ return 0 2023-08-26T14:01:06.449 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-08-26T14:01:06.449 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ image=image_43 2023-08-26T14:01:06.449 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ remove_snapshot cluster2 mirror image_43 snap 2023-08-26T14:01:06.450 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster2 2023-08-26T14:01:06.450 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T14:01:06.450 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=image_43 2023-08-26T14:01:06.450 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local snap=snap 2023-08-26T14:01:06.450 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rbd --cluster cluster2 snap rm mirror/image_43@snap 2023-08-26T14:01:07.312 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr: Removing snap: 100% complete...done. 2023-08-26T14:01:07.320 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ remove_image_retry cluster2 mirror image_43 2023-08-26T14:01:07.320 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster2 2023-08-26T14:01:07.321 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T14:01:07.321 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=image_43 2023-08-26T14:01:07.321 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ for s in 0 1 2 4 8 16 32 2023-08-26T14:01:07.321 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ sleep 0 2023-08-26T14:01:07.322 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ remove_image cluster2 mirror image_43 2023-08-26T14:01:07.322 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster2 2023-08-26T14:01:07.323 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T14:01:07.323 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=image_43 2023-08-26T14:01:07.323 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_43 2023-08-26T14:01:07.365 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rbd --cluster=cluster2 rm mirror/image_43 2023-08-26T14:01:07.474 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi124.stderr:2023-08-26T14:01:07.473+0000 7f6c3363a700 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x5621ba7da9c0 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2023-08-26T14:01:07.474 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi124.stderr:2023-08-26T14:01:07.473+0000 7f6c39e47700 -1 rbd::mirror::ImageReplayer: 0x5621ba457400 [3/93289c2f-b179-4509-b3aa-d0fd920c15e7] operator(): start failed: (85) Interrupted system call should be restarted 2023-08-26T14:01:08.441 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2023-08-26T14:01:08.444 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ return 0 2023-08-26T14:01:08.445 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-08-26T14:01:08.445 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ image=image_44 2023-08-26T14:01:08.445 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ remove_snapshot cluster2 mirror image_44 snap 2023-08-26T14:01:08.445 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster2 2023-08-26T14:01:08.445 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T14:01:08.446 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=image_44 2023-08-26T14:01:08.446 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local snap=snap 2023-08-26T14:01:08.446 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rbd --cluster cluster2 snap rm mirror/image_44@snap 2023-08-26T14:01:09.244 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr: Removing snap: 100% complete...done. 2023-08-26T14:01:09.256 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ remove_image_retry cluster2 mirror image_44 2023-08-26T14:01:09.256 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster2 2023-08-26T14:01:09.256 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T14:01:09.256 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=image_44 2023-08-26T14:01:09.256 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ for s in 0 1 2 4 8 16 32 2023-08-26T14:01:09.257 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ sleep 0 2023-08-26T14:01:09.257 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ remove_image cluster2 mirror image_44 2023-08-26T14:01:09.258 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster2 2023-08-26T14:01:09.258 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T14:01:09.258 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=image_44 2023-08-26T14:01:09.258 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_44 2023-08-26T14:01:09.303 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rbd --cluster=cluster2 rm mirror/image_44 2023-08-26T14:01:09.441 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi124.stderr:2023-08-26T14:01:09.439+0000 7fc244bc9700 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x555d5b94b040 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2023-08-26T14:01:09.441 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi124.stderr:2023-08-26T14:01:09.439+0000 7fc24bbd7700 -1 rbd::mirror::ImageReplayer: 0x555d5ae9b680 [3/11ee1779-6c0f-4f0f-8432-721939d3a164] operator(): start failed: (85) Interrupted system call should be restarted 2023-08-26T14:01:10.452 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2023-08-26T14:01:10.456 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ return 0 2023-08-26T14:01:10.456 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-08-26T14:01:10.457 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ image=image_45 2023-08-26T14:01:10.457 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ remove_snapshot cluster2 mirror image_45 snap 2023-08-26T14:01:10.457 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster2 2023-08-26T14:01:10.458 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T14:01:10.458 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=image_45 2023-08-26T14:01:10.458 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local snap=snap 2023-08-26T14:01:10.459 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rbd --cluster cluster2 snap rm mirror/image_45@snap 2023-08-26T14:01:11.324 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr: Removing snap: 100% complete...done. 2023-08-26T14:01:11.335 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ remove_image_retry cluster2 mirror image_45 2023-08-26T14:01:11.335 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster2 2023-08-26T14:01:11.336 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T14:01:11.336 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=image_45 2023-08-26T14:01:11.336 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ for s in 0 1 2 4 8 16 32 2023-08-26T14:01:11.336 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ sleep 0 2023-08-26T14:01:11.337 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ remove_image cluster2 mirror image_45 2023-08-26T14:01:11.337 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster2 2023-08-26T14:01:11.338 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T14:01:11.338 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=image_45 2023-08-26T14:01:11.338 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_45 2023-08-26T14:01:11.384 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rbd --cluster=cluster2 rm mirror/image_45 2023-08-26T14:01:11.504 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi124.stderr:2023-08-26T14:01:11.503+0000 7fcd435b0700 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x563b722d8340 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2023-08-26T14:01:11.504 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi124.stderr:2023-08-26T14:01:11.503+0000 7fcd495bc700 -1 rbd::mirror::ImageReplayer: 0x563b71eed900 [3/1822564f-3eee-4ba8-b76e-97584ef869b2] operator(): start failed: (85) Interrupted system call should be restarted 2023-08-26T14:01:11.773 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.0 with skip (0.7721487874394665) > weight (0.25) 2023-08-26T14:01:11.773 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.1 with skip (0.6834186367798931) > weight (0.25) 2023-08-26T14:01:11.774 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.2 with skip (0.8325428705115421) > weight (0.25) 2023-08-26T14:01:11.774 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.3 with skip (0.8101517294965012) > weight (0.25) 2023-08-26T14:01:11.774 INFO:tasks.rbd_mirror_thrash:waiting for 115 secs before thrashing 2023-08-26T14:01:12.458 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2023-08-26T14:01:12.462 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ return 0 2023-08-26T14:01:12.462 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-08-26T14:01:12.462 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ image=image_46 2023-08-26T14:01:12.463 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ remove_snapshot cluster2 mirror image_46 snap 2023-08-26T14:01:12.463 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster2 2023-08-26T14:01:12.463 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T14:01:12.463 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=image_46 2023-08-26T14:01:12.463 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local snap=snap 2023-08-26T14:01:12.464 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rbd --cluster cluster2 snap rm mirror/image_46@snap 2023-08-26T14:01:13.324 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr: Removing snap: 100% complete...done. 2023-08-26T14:01:13.334 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ remove_image_retry cluster2 mirror image_46 2023-08-26T14:01:13.334 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster2 2023-08-26T14:01:13.334 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T14:01:13.335 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=image_46 2023-08-26T14:01:13.335 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ for s in 0 1 2 4 8 16 32 2023-08-26T14:01:13.335 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ sleep 0 2023-08-26T14:01:13.337 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ remove_image cluster2 mirror image_46 2023-08-26T14:01:13.337 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster2 2023-08-26T14:01:13.337 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T14:01:13.338 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=image_46 2023-08-26T14:01:13.338 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_46 2023-08-26T14:01:13.387 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rbd --cluster=cluster2 rm mirror/image_46 2023-08-26T14:01:13.499 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi124.stderr:2023-08-26T14:01:13.498+0000 7fc2453ca700 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x555d5b94aea0 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2023-08-26T14:01:13.499 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi124.stderr:2023-08-26T14:01:13.498+0000 7fc24bbd7700 -1 rbd::mirror::ImageReplayer: 0x555d5b991400 [3/994498aa-fae7-419f-8291-f66b876aef55] operator(): start failed: (85) Interrupted system call should be restarted 2023-08-26T14:01:14.363 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2023-08-26T14:01:14.366 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ return 0 2023-08-26T14:01:14.366 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-08-26T14:01:14.367 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ image=image_47 2023-08-26T14:01:14.367 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ remove_snapshot cluster2 mirror image_47 snap 2023-08-26T14:01:14.367 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster2 2023-08-26T14:01:14.367 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T14:01:14.367 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=image_47 2023-08-26T14:01:14.368 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local snap=snap 2023-08-26T14:01:14.368 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rbd --cluster cluster2 snap rm mirror/image_47@snap 2023-08-26T14:01:15.339 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr: Removing snap: 100% complete...done. 2023-08-26T14:01:15.351 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ remove_image_retry cluster2 mirror image_47 2023-08-26T14:01:15.352 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster2 2023-08-26T14:01:15.352 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T14:01:15.352 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=image_47 2023-08-26T14:01:15.352 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ for s in 0 1 2 4 8 16 32 2023-08-26T14:01:15.352 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ sleep 0 2023-08-26T14:01:15.353 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ remove_image cluster2 mirror image_47 2023-08-26T14:01:15.353 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster2 2023-08-26T14:01:15.354 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T14:01:15.354 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=image_47 2023-08-26T14:01:15.354 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_47 2023-08-26T14:01:15.397 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rbd --cluster=cluster2 rm mirror/image_47 2023-08-26T14:01:15.511 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi124.stderr:2023-08-26T14:01:15.510+0000 7fc2453ca700 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x555d5bf56ea0 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2023-08-26T14:01:15.511 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi124.stderr:2023-08-26T14:01:15.510+0000 7fc24b3d6700 -1 rbd::mirror::ImageReplayer: 0x555d5ae9b400 [3/530e7abb-9a3e-4245-a312-8ee5a47a865b] operator(): start failed: (85) Interrupted system call should be restarted 2023-08-26T14:01:16.472 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2023-08-26T14:01:16.475 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ return 0 2023-08-26T14:01:16.476 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-08-26T14:01:16.476 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ image=image_48 2023-08-26T14:01:16.476 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ remove_snapshot cluster2 mirror image_48 snap 2023-08-26T14:01:16.476 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster2 2023-08-26T14:01:16.477 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T14:01:16.477 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=image_48 2023-08-26T14:01:16.477 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local snap=snap 2023-08-26T14:01:16.477 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rbd --cluster cluster2 snap rm mirror/image_48@snap 2023-08-26T14:01:17.344 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr: Removing snap: 100% complete...done. 2023-08-26T14:01:17.358 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ remove_image_retry cluster2 mirror image_48 2023-08-26T14:01:17.358 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster2 2023-08-26T14:01:17.359 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T14:01:17.359 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=image_48 2023-08-26T14:01:17.359 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ for s in 0 1 2 4 8 16 32 2023-08-26T14:01:17.359 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ sleep 0 2023-08-26T14:01:17.360 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ remove_image cluster2 mirror image_48 2023-08-26T14:01:17.360 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster2 2023-08-26T14:01:17.360 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T14:01:17.361 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=image_48 2023-08-26T14:01:17.361 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_48 2023-08-26T14:01:17.421 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rbd --cluster=cluster2 rm mirror/image_48 2023-08-26T14:01:17.542 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi124.stderr:2023-08-26T14:01:17.541+0000 7fc0c3936700 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x5567e0dda680 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2023-08-26T14:01:17.543 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi124.stderr:2023-08-26T14:01:17.541+0000 7fc0ca143700 -1 rbd::mirror::ImageReplayer: 0x5567e030cc80 [3/6721bd5f-42ff-4eab-84c6-4ca33fff75c0] operator(): start failed: (85) Interrupted system call should be restarted 2023-08-26T14:01:18.426 INFO:tasks.ceph.cluster2.osd.2.smithi124.stderr:2023-08-26T14:01:18.424+0000 7fb790854700 -1 reset not still connected to 0x5598e1a43a00 2023-08-26T14:01:18.488 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2023-08-26T14:01:18.491 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ return 0 2023-08-26T14:01:18.492 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-08-26T14:01:18.492 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ image=image_49 2023-08-26T14:01:18.492 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ remove_snapshot cluster2 mirror image_49 snap 2023-08-26T14:01:18.492 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster2 2023-08-26T14:01:18.492 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T14:01:18.493 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=image_49 2023-08-26T14:01:18.493 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local snap=snap 2023-08-26T14:01:18.493 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rbd --cluster cluster2 snap rm mirror/image_49@snap 2023-08-26T14:01:19.253 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr: Removing snap: 100% complete...done. 2023-08-26T14:01:19.262 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ remove_image_retry cluster2 mirror image_49 2023-08-26T14:01:19.262 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster2 2023-08-26T14:01:19.262 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T14:01:19.263 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=image_49 2023-08-26T14:01:19.263 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ for s in 0 1 2 4 8 16 32 2023-08-26T14:01:19.263 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ sleep 0 2023-08-26T14:01:19.264 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ remove_image cluster2 mirror image_49 2023-08-26T14:01:19.264 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster2 2023-08-26T14:01:19.264 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T14:01:19.264 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=image_49 2023-08-26T14:01:19.265 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_49 2023-08-26T14:01:19.312 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rbd --cluster=cluster2 rm mirror/image_49 2023-08-26T14:01:19.449 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi124.stderr:2023-08-26T14:01:19.448+0000 7fc0c3936700 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x5567e0dda820 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2023-08-26T14:01:19.450 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi124.stderr:2023-08-26T14:01:19.448+0000 7fc0ca143700 -1 rbd::mirror::ImageReplayer: 0x5567e0ba2780 [3/5f1ee9ae-1ac1-4156-b437-c09951a7762e] operator(): start failed: (85) Interrupted system call should be restarted 2023-08-26T14:01:20.485 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2023-08-26T14:01:20.488 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ return 0 2023-08-26T14:01:20.489 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-08-26T14:01:20.489 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ image=image_50 2023-08-26T14:01:20.489 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ remove_snapshot cluster2 mirror image_50 snap 2023-08-26T14:01:20.489 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster2 2023-08-26T14:01:20.489 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T14:01:20.490 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=image_50 2023-08-26T14:01:20.490 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local snap=snap 2023-08-26T14:01:20.490 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rbd --cluster cluster2 snap rm mirror/image_50@snap 2023-08-26T14:01:21.346 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr: Removing snap: 100% complete...done. 2023-08-26T14:01:21.357 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ remove_image_retry cluster2 mirror image_50 2023-08-26T14:01:21.358 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster2 2023-08-26T14:01:21.358 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T14:01:21.358 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=image_50 2023-08-26T14:01:21.358 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ for s in 0 1 2 4 8 16 32 2023-08-26T14:01:21.358 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ sleep 0 2023-08-26T14:01:21.359 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ remove_image cluster2 mirror image_50 2023-08-26T14:01:21.359 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster2 2023-08-26T14:01:21.359 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T14:01:21.359 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=image_50 2023-08-26T14:01:21.360 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_50 2023-08-26T14:01:21.410 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rbd --cluster=cluster2 rm mirror/image_50 2023-08-26T14:01:21.534 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi124.stderr:2023-08-26T14:01:21.533+0000 7fcd435b0700 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x563b72260340 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2023-08-26T14:01:21.534 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi124.stderr:2023-08-26T14:01:21.533+0000 7fcd49dbd700 -1 rbd::mirror::ImageReplayer: 0x563b71eecf00 [3/ebb4d874-359b-4be6-9915-df44e455c46f] operator(): start failed: (85) Interrupted system call should be restarted 2023-08-26T14:01:22.496 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2023-08-26T14:01:22.499 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ return 0 2023-08-26T14:01:22.499 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ testlog 'TEST: image deletions should propagate' 2023-08-26T14:01:22.500 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ tee -a /tmp/tmp.TfDn6BcQ3N/rbd-mirror.test.log 2023-08-26T14:01:22.500 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ date '+%F %T' 2023-08-26T14:01:22.502 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ echo 2023-08-26 14:01:22 TEST: image deletions should propagate 2023-08-26T14:01:22.502 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:2023-08-26 14:01:22 TEST: image deletions should propagate 2023-08-26T14:01:22.502 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ wait_for_pool_images cluster1 mirror 0 2023-08-26T14:01:22.502 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster1 2023-08-26T14:01:22.503 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T14:01:22.503 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image_count=0 2023-08-26T14:01:22.503 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local s 2023-08-26T14:01:22.503 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local count 2023-08-26T14:01:22.503 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local last_count=0 2023-08-26T14:01:22.504 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ true 2023-08-26T14:01:22.504 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ seq 1 40 2023-08-26T14:01:22.504 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ for s in `seq 1 40` 2023-08-26T14:01:22.504 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ test 1 -ne 1 2023-08-26T14:01:22.504 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ rbd --cluster cluster1 -p mirror mirror pool status 2023-08-26T14:01:22.505 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ grep 'images: ' 2023-08-26T14:01:22.505 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ cut '-d ' -f 2 2023-08-26T14:01:22.535 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:rbd: failed to query services: (13) Permission denied 2023-08-26T14:01:22.538 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ count=0 2023-08-26T14:01:22.539 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ test 0 = 0 2023-08-26T14:01:22.539 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ return 0 2023-08-26T14:01:22.539 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ wait_for_pool_healthy cluster1 mirror 0 2023-08-26T14:01:22.539 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster1 2023-08-26T14:01:22.540 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T14:01:22.540 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local s 2023-08-26T14:01:22.540 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local state 2023-08-26T14:01:22.540 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ seq 1 40 2023-08-26T14:01:22.541 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ for s in `seq 1 40` 2023-08-26T14:01:22.541 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ test 1 -ne 1 2023-08-26T14:01:22.541 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ rbd --cluster cluster1 -p mirror mirror pool status 2023-08-26T14:01:22.542 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ grep 'image health:' 2023-08-26T14:01:22.542 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ cut '-d ' -f 3 2023-08-26T14:01:22.572 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:rbd: failed to query services: (13) Permission denied 2023-08-26T14:01:22.576 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ state=OK 2023-08-26T14:01:22.576 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ test OK = ERROR 2023-08-26T14:01:22.576 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ test OK = OK 2023-08-26T14:01:22.576 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ return 0 2023-08-26T14:01:22.577 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ seq 1 50 2023-08-26T14:01:22.578 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-08-26T14:01:22.578 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ image=image_1 2023-08-26T14:01:22.578 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ wait_for_image_present cluster1 mirror image_1 deleted 2023-08-26T14:01:22.579 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster1 2023-08-26T14:01:22.579 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T14:01:22.579 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=image_1 2023-08-26T14:01:22.579 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local state=deleted 2023-08-26T14:01:22.579 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image_id= 2023-08-26T14:01:22.580 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local s 2023-08-26T14:01:22.580 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ test -n '' 2023-08-26T14:01:22.580 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ get_image_id cluster1 mirror image_1 2023-08-26T14:01:22.580 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local cluster=cluster1 2023-08-26T14:01:22.580 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local pool=mirror 2023-08-26T14:01:22.580 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local image=image_1 2023-08-26T14:01:22.581 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ rbd --cluster=cluster1 info mirror/image_1 2023-08-26T14:01:22.581 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2023-08-26T14:01:22.609 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:rbd: error opening image image_1: (2) No such file or directory 2023-08-26T14:01:22.611 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ image_id= 2023-08-26T14:01:22.612 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2023-08-26T14:01:22.612 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ sleep 0.1 2023-08-26T14:01:22.713 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ test_image_present cluster1 mirror image_1 deleted '' 2023-08-26T14:01:22.714 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster1 2023-08-26T14:01:22.714 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T14:01:22.714 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=image_1 2023-08-26T14:01:22.714 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local test_state=deleted 2023-08-26T14:01:22.714 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image_id= 2023-08-26T14:01:22.715 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local current_state=deleted 2023-08-26T14:01:22.715 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local current_image_id 2023-08-26T14:01:22.715 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ get_image_id cluster1 mirror image_1 2023-08-26T14:01:22.715 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local cluster=cluster1 2023-08-26T14:01:22.715 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local pool=mirror 2023-08-26T14:01:22.716 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local image=image_1 2023-08-26T14:01:22.716 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ rbd --cluster=cluster1 info mirror/image_1 2023-08-26T14:01:22.717 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2023-08-26T14:01:22.745 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:rbd: error opening image image_1: (2) No such file or directory 2023-08-26T14:01:22.749 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ current_image_id= 2023-08-26T14:01:22.749 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ test -n '' 2023-08-26T14:01:22.749 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ test deleted = deleted 2023-08-26T14:01:22.749 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ return 0 2023-08-26T14:01:22.750 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-08-26T14:01:22.750 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ image=image_2 2023-08-26T14:01:22.750 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ wait_for_image_present cluster1 mirror image_2 deleted 2023-08-26T14:01:22.750 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster1 2023-08-26T14:01:22.750 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T14:01:22.751 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=image_2 2023-08-26T14:01:22.751 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local state=deleted 2023-08-26T14:01:22.751 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image_id= 2023-08-26T14:01:22.751 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local s 2023-08-26T14:01:22.751 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ test -n '' 2023-08-26T14:01:22.752 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ get_image_id cluster1 mirror image_2 2023-08-26T14:01:22.752 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local cluster=cluster1 2023-08-26T14:01:22.752 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local pool=mirror 2023-08-26T14:01:22.752 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local image=image_2 2023-08-26T14:01:22.752 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ rbd --cluster=cluster1 info mirror/image_2 2023-08-26T14:01:22.753 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2023-08-26T14:01:22.781 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:rbd: error opening image image_2: (2) No such file or directory 2023-08-26T14:01:22.784 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ image_id= 2023-08-26T14:01:22.784 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2023-08-26T14:01:22.784 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ sleep 0.1 2023-08-26T14:01:22.886 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ test_image_present cluster1 mirror image_2 deleted '' 2023-08-26T14:01:22.886 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster1 2023-08-26T14:01:22.886 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T14:01:22.886 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=image_2 2023-08-26T14:01:22.887 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local test_state=deleted 2023-08-26T14:01:22.887 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image_id= 2023-08-26T14:01:22.887 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local current_state=deleted 2023-08-26T14:01:22.887 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local current_image_id 2023-08-26T14:01:22.887 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ get_image_id cluster1 mirror image_2 2023-08-26T14:01:22.888 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local cluster=cluster1 2023-08-26T14:01:22.888 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local pool=mirror 2023-08-26T14:01:22.888 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local image=image_2 2023-08-26T14:01:22.888 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ rbd --cluster=cluster1 info mirror/image_2 2023-08-26T14:01:22.888 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2023-08-26T14:01:22.918 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:rbd: error opening image image_2: (2) No such file or directory 2023-08-26T14:01:22.922 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ current_image_id= 2023-08-26T14:01:22.922 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ test -n '' 2023-08-26T14:01:22.922 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ test deleted = deleted 2023-08-26T14:01:22.923 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ return 0 2023-08-26T14:01:22.923 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-08-26T14:01:22.923 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ image=image_3 2023-08-26T14:01:22.923 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ wait_for_image_present cluster1 mirror image_3 deleted 2023-08-26T14:01:22.923 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster1 2023-08-26T14:01:22.924 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T14:01:22.924 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=image_3 2023-08-26T14:01:22.924 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local state=deleted 2023-08-26T14:01:22.925 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image_id= 2023-08-26T14:01:22.925 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local s 2023-08-26T14:01:22.925 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ test -n '' 2023-08-26T14:01:22.925 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ get_image_id cluster1 mirror image_3 2023-08-26T14:01:22.926 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local cluster=cluster1 2023-08-26T14:01:22.926 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local pool=mirror 2023-08-26T14:01:22.926 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local image=image_3 2023-08-26T14:01:22.926 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ rbd --cluster=cluster1 info mirror/image_3 2023-08-26T14:01:22.926 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2023-08-26T14:01:23.155 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:rbd: error opening image image_3: (2) No such file or directory 2023-08-26T14:01:23.158 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ image_id= 2023-08-26T14:01:23.159 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2023-08-26T14:01:23.159 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ sleep 0.1 2023-08-26T14:01:23.260 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ test_image_present cluster1 mirror image_3 deleted '' 2023-08-26T14:01:23.261 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster1 2023-08-26T14:01:23.261 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T14:01:23.261 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=image_3 2023-08-26T14:01:23.261 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local test_state=deleted 2023-08-26T14:01:23.261 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image_id= 2023-08-26T14:01:23.262 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local current_state=deleted 2023-08-26T14:01:23.262 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local current_image_id 2023-08-26T14:01:23.262 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ get_image_id cluster1 mirror image_3 2023-08-26T14:01:23.262 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local cluster=cluster1 2023-08-26T14:01:23.262 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local pool=mirror 2023-08-26T14:01:23.263 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local image=image_3 2023-08-26T14:01:23.263 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ rbd --cluster=cluster1 info mirror/image_3 2023-08-26T14:01:23.263 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2023-08-26T14:01:23.293 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:rbd: error opening image image_3: (2) No such file or directory 2023-08-26T14:01:23.297 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ current_image_id= 2023-08-26T14:01:23.297 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ test -n '' 2023-08-26T14:01:23.297 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ test deleted = deleted 2023-08-26T14:01:23.298 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ return 0 2023-08-26T14:01:23.298 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-08-26T14:01:23.298 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ image=image_4 2023-08-26T14:01:23.298 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ wait_for_image_present cluster1 mirror image_4 deleted 2023-08-26T14:01:23.298 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster1 2023-08-26T14:01:23.299 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T14:01:23.299 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=image_4 2023-08-26T14:01:23.299 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local state=deleted 2023-08-26T14:01:23.299 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image_id= 2023-08-26T14:01:23.299 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local s 2023-08-26T14:01:23.300 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ test -n '' 2023-08-26T14:01:23.300 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ get_image_id cluster1 mirror image_4 2023-08-26T14:01:23.300 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local cluster=cluster1 2023-08-26T14:01:23.300 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local pool=mirror 2023-08-26T14:01:23.300 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local image=image_4 2023-08-26T14:01:23.300 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ rbd --cluster=cluster1 info mirror/image_4 2023-08-26T14:01:23.301 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2023-08-26T14:01:23.329 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:rbd: error opening image image_4: (2) No such file or directory 2023-08-26T14:01:23.332 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ image_id= 2023-08-26T14:01:23.332 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2023-08-26T14:01:23.332 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ sleep 0.1 2023-08-26T14:01:23.434 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ test_image_present cluster1 mirror image_4 deleted '' 2023-08-26T14:01:23.434 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster1 2023-08-26T14:01:23.435 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T14:01:23.435 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=image_4 2023-08-26T14:01:23.435 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local test_state=deleted 2023-08-26T14:01:23.435 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image_id= 2023-08-26T14:01:23.435 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local current_state=deleted 2023-08-26T14:01:23.436 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local current_image_id 2023-08-26T14:01:23.436 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ get_image_id cluster1 mirror image_4 2023-08-26T14:01:23.436 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local cluster=cluster1 2023-08-26T14:01:23.436 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local pool=mirror 2023-08-26T14:01:23.436 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local image=image_4 2023-08-26T14:01:23.437 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ rbd --cluster=cluster1 info mirror/image_4 2023-08-26T14:01:23.437 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2023-08-26T14:01:23.470 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:rbd: error opening image image_4: (2) No such file or directory 2023-08-26T14:01:23.474 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ current_image_id= 2023-08-26T14:01:23.474 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ test -n '' 2023-08-26T14:01:23.474 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ test deleted = deleted 2023-08-26T14:01:23.474 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ return 0 2023-08-26T14:01:23.474 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-08-26T14:01:23.475 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ image=image_5 2023-08-26T14:01:23.475 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ wait_for_image_present cluster1 mirror image_5 deleted 2023-08-26T14:01:23.475 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster1 2023-08-26T14:01:23.475 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T14:01:23.475 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=image_5 2023-08-26T14:01:23.476 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local state=deleted 2023-08-26T14:01:23.476 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image_id= 2023-08-26T14:01:23.476 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local s 2023-08-26T14:01:23.476 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ test -n '' 2023-08-26T14:01:23.476 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ get_image_id cluster1 mirror image_5 2023-08-26T14:01:23.476 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local cluster=cluster1 2023-08-26T14:01:23.477 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local pool=mirror 2023-08-26T14:01:23.477 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local image=image_5 2023-08-26T14:01:23.477 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ rbd --cluster=cluster1 info mirror/image_5 2023-08-26T14:01:23.477 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2023-08-26T14:01:23.708 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:rbd: error opening image image_5: (2) No such file or directory 2023-08-26T14:01:23.711 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ image_id= 2023-08-26T14:01:23.711 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2023-08-26T14:01:23.712 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ sleep 0.1 2023-08-26T14:01:23.813 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ test_image_present cluster1 mirror image_5 deleted '' 2023-08-26T14:01:23.813 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster1 2023-08-26T14:01:23.813 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T14:01:23.814 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=image_5 2023-08-26T14:01:23.814 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local test_state=deleted 2023-08-26T14:01:23.814 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image_id= 2023-08-26T14:01:23.814 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local current_state=deleted 2023-08-26T14:01:23.814 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local current_image_id 2023-08-26T14:01:23.815 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ get_image_id cluster1 mirror image_5 2023-08-26T14:01:23.815 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local cluster=cluster1 2023-08-26T14:01:23.815 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local pool=mirror 2023-08-26T14:01:23.815 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local image=image_5 2023-08-26T14:01:23.815 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ rbd --cluster=cluster1 info mirror/image_5 2023-08-26T14:01:23.816 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2023-08-26T14:01:23.846 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:rbd: error opening image image_5: (2) No such file or directory 2023-08-26T14:01:23.849 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ current_image_id= 2023-08-26T14:01:23.850 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ test -n '' 2023-08-26T14:01:23.850 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ test deleted = deleted 2023-08-26T14:01:23.850 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ return 0 2023-08-26T14:01:23.850 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-08-26T14:01:23.850 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ image=image_6 2023-08-26T14:01:23.851 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ wait_for_image_present cluster1 mirror image_6 deleted 2023-08-26T14:01:23.851 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster1 2023-08-26T14:01:23.851 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T14:01:23.851 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=image_6 2023-08-26T14:01:23.851 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local state=deleted 2023-08-26T14:01:23.852 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image_id= 2023-08-26T14:01:23.852 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local s 2023-08-26T14:01:23.852 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ test -n '' 2023-08-26T14:01:23.852 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ get_image_id cluster1 mirror image_6 2023-08-26T14:01:23.852 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local cluster=cluster1 2023-08-26T14:01:23.853 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local pool=mirror 2023-08-26T14:01:23.853 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local image=image_6 2023-08-26T14:01:23.853 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ rbd --cluster=cluster1 info mirror/image_6 2023-08-26T14:01:23.853 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2023-08-26T14:01:24.082 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:rbd: error opening image image_6: (2) No such file or directory 2023-08-26T14:01:24.085 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ image_id= 2023-08-26T14:01:24.085 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2023-08-26T14:01:24.085 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ sleep 0.1 2023-08-26T14:01:24.187 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ test_image_present cluster1 mirror image_6 deleted '' 2023-08-26T14:01:24.187 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster1 2023-08-26T14:01:24.188 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T14:01:24.188 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=image_6 2023-08-26T14:01:24.188 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local test_state=deleted 2023-08-26T14:01:24.188 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image_id= 2023-08-26T14:01:24.188 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local current_state=deleted 2023-08-26T14:01:24.189 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local current_image_id 2023-08-26T14:01:24.189 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ get_image_id cluster1 mirror image_6 2023-08-26T14:01:24.189 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local cluster=cluster1 2023-08-26T14:01:24.189 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local pool=mirror 2023-08-26T14:01:24.189 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local image=image_6 2023-08-26T14:01:24.190 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ rbd --cluster=cluster1 info mirror/image_6 2023-08-26T14:01:24.190 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2023-08-26T14:01:24.220 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:rbd: error opening image image_6: (2) No such file or directory 2023-08-26T14:01:24.225 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ current_image_id= 2023-08-26T14:01:24.225 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ test -n '' 2023-08-26T14:01:24.226 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ test deleted = deleted 2023-08-26T14:01:24.226 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ return 0 2023-08-26T14:01:24.226 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-08-26T14:01:24.226 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ image=image_7 2023-08-26T14:01:24.226 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ wait_for_image_present cluster1 mirror image_7 deleted 2023-08-26T14:01:24.226 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster1 2023-08-26T14:01:24.227 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T14:01:24.227 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=image_7 2023-08-26T14:01:24.227 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local state=deleted 2023-08-26T14:01:24.227 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image_id= 2023-08-26T14:01:24.227 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local s 2023-08-26T14:01:24.227 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ test -n '' 2023-08-26T14:01:24.228 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ get_image_id cluster1 mirror image_7 2023-08-26T14:01:24.228 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local cluster=cluster1 2023-08-26T14:01:24.228 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local pool=mirror 2023-08-26T14:01:24.228 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local image=image_7 2023-08-26T14:01:24.228 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ rbd --cluster=cluster1 info mirror/image_7 2023-08-26T14:01:24.229 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2023-08-26T14:01:24.256 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:rbd: error opening image image_7: (2) No such file or directory 2023-08-26T14:01:24.260 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ image_id= 2023-08-26T14:01:24.260 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2023-08-26T14:01:24.261 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ sleep 0.1 2023-08-26T14:01:24.361 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ test_image_present cluster1 mirror image_7 deleted '' 2023-08-26T14:01:24.361 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster1 2023-08-26T14:01:24.361 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T14:01:24.361 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=image_7 2023-08-26T14:01:24.362 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local test_state=deleted 2023-08-26T14:01:24.362 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image_id= 2023-08-26T14:01:24.362 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local current_state=deleted 2023-08-26T14:01:24.362 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local current_image_id 2023-08-26T14:01:24.362 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ get_image_id cluster1 mirror image_7 2023-08-26T14:01:24.362 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local cluster=cluster1 2023-08-26T14:01:24.363 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local pool=mirror 2023-08-26T14:01:24.363 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local image=image_7 2023-08-26T14:01:24.363 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ rbd --cluster=cluster1 info mirror/image_7 2023-08-26T14:01:24.363 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2023-08-26T14:01:24.393 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:rbd: error opening image image_7: (2) No such file or directory 2023-08-26T14:01:24.397 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ current_image_id= 2023-08-26T14:01:24.397 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ test -n '' 2023-08-26T14:01:24.398 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ test deleted = deleted 2023-08-26T14:01:24.398 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ return 0 2023-08-26T14:01:24.398 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-08-26T14:01:24.398 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ image=image_8 2023-08-26T14:01:24.398 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ wait_for_image_present cluster1 mirror image_8 deleted 2023-08-26T14:01:24.398 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster1 2023-08-26T14:01:24.399 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T14:01:24.399 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=image_8 2023-08-26T14:01:24.399 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local state=deleted 2023-08-26T14:01:24.399 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image_id= 2023-08-26T14:01:24.399 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local s 2023-08-26T14:01:24.400 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ test -n '' 2023-08-26T14:01:24.400 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ get_image_id cluster1 mirror image_8 2023-08-26T14:01:24.400 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local cluster=cluster1 2023-08-26T14:01:24.400 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local pool=mirror 2023-08-26T14:01:24.401 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local image=image_8 2023-08-26T14:01:24.401 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ rbd --cluster=cluster1 info mirror/image_8 2023-08-26T14:01:24.401 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2023-08-26T14:01:24.446 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:rbd: error opening image image_8: (2) No such file or directory 2023-08-26T14:01:24.449 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ image_id= 2023-08-26T14:01:24.449 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2023-08-26T14:01:24.449 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ sleep 0.1 2023-08-26T14:01:24.551 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ test_image_present cluster1 mirror image_8 deleted '' 2023-08-26T14:01:24.551 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster1 2023-08-26T14:01:24.551 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T14:01:24.551 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=image_8 2023-08-26T14:01:24.552 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local test_state=deleted 2023-08-26T14:01:24.552 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image_id= 2023-08-26T14:01:24.552 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local current_state=deleted 2023-08-26T14:01:24.552 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local current_image_id 2023-08-26T14:01:24.552 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ get_image_id cluster1 mirror image_8 2023-08-26T14:01:24.553 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local cluster=cluster1 2023-08-26T14:01:24.553 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local pool=mirror 2023-08-26T14:01:24.553 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local image=image_8 2023-08-26T14:01:24.553 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ rbd --cluster=cluster1 info mirror/image_8 2023-08-26T14:01:24.553 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2023-08-26T14:01:24.583 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:rbd: error opening image image_8: (2) No such file or directory 2023-08-26T14:01:24.588 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ current_image_id= 2023-08-26T14:01:24.588 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ test -n '' 2023-08-26T14:01:24.588 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ test deleted = deleted 2023-08-26T14:01:24.588 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ return 0 2023-08-26T14:01:24.589 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-08-26T14:01:24.589 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ image=image_9 2023-08-26T14:01:24.589 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ wait_for_image_present cluster1 mirror image_9 deleted 2023-08-26T14:01:24.589 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster1 2023-08-26T14:01:24.589 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T14:01:24.589 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=image_9 2023-08-26T14:01:24.590 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local state=deleted 2023-08-26T14:01:24.590 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image_id= 2023-08-26T14:01:24.590 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local s 2023-08-26T14:01:24.590 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ test -n '' 2023-08-26T14:01:24.590 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ get_image_id cluster1 mirror image_9 2023-08-26T14:01:24.591 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local cluster=cluster1 2023-08-26T14:01:24.591 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local pool=mirror 2023-08-26T14:01:24.591 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local image=image_9 2023-08-26T14:01:24.591 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ rbd --cluster=cluster1 info mirror/image_9 2023-08-26T14:01:24.591 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2023-08-26T14:01:24.820 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:rbd: error opening image image_9: (2) No such file or directory 2023-08-26T14:01:24.823 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ image_id= 2023-08-26T14:01:24.823 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2023-08-26T14:01:24.823 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ sleep 0.1 2023-08-26T14:01:24.924 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ test_image_present cluster1 mirror image_9 deleted '' 2023-08-26T14:01:24.925 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster1 2023-08-26T14:01:24.925 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T14:01:24.925 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=image_9 2023-08-26T14:01:24.925 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local test_state=deleted 2023-08-26T14:01:24.925 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image_id= 2023-08-26T14:01:24.926 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local current_state=deleted 2023-08-26T14:01:24.926 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local current_image_id 2023-08-26T14:01:24.926 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ get_image_id cluster1 mirror image_9 2023-08-26T14:01:24.926 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local cluster=cluster1 2023-08-26T14:01:24.926 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local pool=mirror 2023-08-26T14:01:24.926 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local image=image_9 2023-08-26T14:01:24.926 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ rbd --cluster=cluster1 info mirror/image_9 2023-08-26T14:01:24.927 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2023-08-26T14:01:24.957 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:rbd: error opening image image_9: (2) No such file or directory 2023-08-26T14:01:24.961 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ current_image_id= 2023-08-26T14:01:24.962 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ test -n '' 2023-08-26T14:01:24.962 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ test deleted = deleted 2023-08-26T14:01:24.962 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ return 0 2023-08-26T14:01:24.962 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-08-26T14:01:24.963 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ image=image_10 2023-08-26T14:01:24.963 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ wait_for_image_present cluster1 mirror image_10 deleted 2023-08-26T14:01:24.963 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster1 2023-08-26T14:01:24.963 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T14:01:24.963 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=image_10 2023-08-26T14:01:24.963 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local state=deleted 2023-08-26T14:01:24.964 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image_id= 2023-08-26T14:01:24.964 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local s 2023-08-26T14:01:24.964 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ test -n '' 2023-08-26T14:01:24.964 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ get_image_id cluster1 mirror image_10 2023-08-26T14:01:24.964 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local cluster=cluster1 2023-08-26T14:01:24.965 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local pool=mirror 2023-08-26T14:01:24.965 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local image=image_10 2023-08-26T14:01:24.965 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ rbd --cluster=cluster1 info mirror/image_10 2023-08-26T14:01:24.965 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2023-08-26T14:01:24.993 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:rbd: error opening image image_10: (2) No such file or directory 2023-08-26T14:01:24.996 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ image_id= 2023-08-26T14:01:24.996 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2023-08-26T14:01:24.996 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ sleep 0.1 2023-08-26T14:01:25.097 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ test_image_present cluster1 mirror image_10 deleted '' 2023-08-26T14:01:25.098 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster1 2023-08-26T14:01:25.098 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T14:01:25.098 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=image_10 2023-08-26T14:01:25.098 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local test_state=deleted 2023-08-26T14:01:25.099 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image_id= 2023-08-26T14:01:25.099 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local current_state=deleted 2023-08-26T14:01:25.099 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local current_image_id 2023-08-26T14:01:25.099 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ get_image_id cluster1 mirror image_10 2023-08-26T14:01:25.099 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local cluster=cluster1 2023-08-26T14:01:25.099 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local pool=mirror 2023-08-26T14:01:25.100 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local image=image_10 2023-08-26T14:01:25.100 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ rbd --cluster=cluster1 info mirror/image_10 2023-08-26T14:01:25.100 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2023-08-26T14:01:25.130 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:rbd: error opening image image_10: (2) No such file or directory 2023-08-26T14:01:25.134 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ current_image_id= 2023-08-26T14:01:25.134 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ test -n '' 2023-08-26T14:01:25.134 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ test deleted = deleted 2023-08-26T14:01:25.134 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ return 0 2023-08-26T14:01:25.134 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-08-26T14:01:25.135 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ image=image_11 2023-08-26T14:01:25.135 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ wait_for_image_present cluster1 mirror image_11 deleted 2023-08-26T14:01:25.135 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster1 2023-08-26T14:01:25.135 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T14:01:25.135 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=image_11 2023-08-26T14:01:25.135 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local state=deleted 2023-08-26T14:01:25.136 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image_id= 2023-08-26T14:01:25.136 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local s 2023-08-26T14:01:25.136 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ test -n '' 2023-08-26T14:01:25.136 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ get_image_id cluster1 mirror image_11 2023-08-26T14:01:25.136 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local cluster=cluster1 2023-08-26T14:01:25.137 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local pool=mirror 2023-08-26T14:01:25.137 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local image=image_11 2023-08-26T14:01:25.137 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ rbd --cluster=cluster1 info mirror/image_11 2023-08-26T14:01:25.137 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2023-08-26T14:01:25.165 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:rbd: error opening image image_11: (2) No such file or directory 2023-08-26T14:01:25.168 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ image_id= 2023-08-26T14:01:25.168 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2023-08-26T14:01:25.168 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ sleep 0.1 2023-08-26T14:01:25.270 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ test_image_present cluster1 mirror image_11 deleted '' 2023-08-26T14:01:25.270 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster1 2023-08-26T14:01:25.270 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T14:01:25.271 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=image_11 2023-08-26T14:01:25.271 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local test_state=deleted 2023-08-26T14:01:25.271 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image_id= 2023-08-26T14:01:25.271 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local current_state=deleted 2023-08-26T14:01:25.271 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local current_image_id 2023-08-26T14:01:25.272 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ get_image_id cluster1 mirror image_11 2023-08-26T14:01:25.272 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local cluster=cluster1 2023-08-26T14:01:25.272 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local pool=mirror 2023-08-26T14:01:25.272 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local image=image_11 2023-08-26T14:01:25.272 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ rbd --cluster=cluster1 info mirror/image_11 2023-08-26T14:01:25.273 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2023-08-26T14:01:25.302 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:rbd: error opening image image_11: (2) No such file or directory 2023-08-26T14:01:25.306 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ current_image_id= 2023-08-26T14:01:25.306 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ test -n '' 2023-08-26T14:01:25.306 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ test deleted = deleted 2023-08-26T14:01:25.307 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ return 0 2023-08-26T14:01:25.307 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-08-26T14:01:25.307 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ image=image_12 2023-08-26T14:01:25.307 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ wait_for_image_present cluster1 mirror image_12 deleted 2023-08-26T14:01:25.308 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster1 2023-08-26T14:01:25.308 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T14:01:25.308 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=image_12 2023-08-26T14:01:25.308 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local state=deleted 2023-08-26T14:01:25.309 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image_id= 2023-08-26T14:01:25.309 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local s 2023-08-26T14:01:25.309 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ test -n '' 2023-08-26T14:01:25.309 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ get_image_id cluster1 mirror image_12 2023-08-26T14:01:25.310 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local cluster=cluster1 2023-08-26T14:01:25.310 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local pool=mirror 2023-08-26T14:01:25.310 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local image=image_12 2023-08-26T14:01:25.310 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ rbd --cluster=cluster1 info mirror/image_12 2023-08-26T14:01:25.310 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2023-08-26T14:01:25.337 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:rbd: error opening image image_12: (2) No such file or directory 2023-08-26T14:01:25.339 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ image_id= 2023-08-26T14:01:25.340 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2023-08-26T14:01:25.340 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ sleep 0.1 2023-08-26T14:01:25.441 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ test_image_present cluster1 mirror image_12 deleted '' 2023-08-26T14:01:25.442 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster1 2023-08-26T14:01:25.442 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T14:01:25.442 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=image_12 2023-08-26T14:01:25.442 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local test_state=deleted 2023-08-26T14:01:25.442 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image_id= 2023-08-26T14:01:25.443 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local current_state=deleted 2023-08-26T14:01:25.443 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local current_image_id 2023-08-26T14:01:25.443 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ get_image_id cluster1 mirror image_12 2023-08-26T14:01:25.443 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local cluster=cluster1 2023-08-26T14:01:25.443 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local pool=mirror 2023-08-26T14:01:25.444 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local image=image_12 2023-08-26T14:01:25.444 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ rbd --cluster=cluster1 info mirror/image_12 2023-08-26T14:01:25.444 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2023-08-26T14:01:25.474 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:rbd: error opening image image_12: (2) No such file or directory 2023-08-26T14:01:25.478 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ current_image_id= 2023-08-26T14:01:25.478 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ test -n '' 2023-08-26T14:01:25.478 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ test deleted = deleted 2023-08-26T14:01:25.478 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ return 0 2023-08-26T14:01:25.478 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-08-26T14:01:25.479 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ image=image_13 2023-08-26T14:01:25.479 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ wait_for_image_present cluster1 mirror image_13 deleted 2023-08-26T14:01:25.479 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster1 2023-08-26T14:01:25.479 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T14:01:25.480 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=image_13 2023-08-26T14:01:25.480 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local state=deleted 2023-08-26T14:01:25.480 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image_id= 2023-08-26T14:01:25.480 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local s 2023-08-26T14:01:25.481 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ test -n '' 2023-08-26T14:01:25.481 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ get_image_id cluster1 mirror image_13 2023-08-26T14:01:25.482 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local cluster=cluster1 2023-08-26T14:01:25.482 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local pool=mirror 2023-08-26T14:01:25.482 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local image=image_13 2023-08-26T14:01:25.483 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ rbd --cluster=cluster1 info mirror/image_13 2023-08-26T14:01:25.483 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2023-08-26T14:01:25.509 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:rbd: error opening image image_13: (2) No such file or directory 2023-08-26T14:01:25.512 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ image_id= 2023-08-26T14:01:25.512 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2023-08-26T14:01:25.512 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ sleep 0.1 2023-08-26T14:01:25.613 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ test_image_present cluster1 mirror image_13 deleted '' 2023-08-26T14:01:25.614 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster1 2023-08-26T14:01:25.614 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T14:01:25.614 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=image_13 2023-08-26T14:01:25.614 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local test_state=deleted 2023-08-26T14:01:25.615 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image_id= 2023-08-26T14:01:25.615 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local current_state=deleted 2023-08-26T14:01:25.615 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local current_image_id 2023-08-26T14:01:25.615 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ get_image_id cluster1 mirror image_13 2023-08-26T14:01:25.615 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local cluster=cluster1 2023-08-26T14:01:25.616 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local pool=mirror 2023-08-26T14:01:25.616 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local image=image_13 2023-08-26T14:01:25.616 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ rbd --cluster=cluster1 info mirror/image_13 2023-08-26T14:01:25.616 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2023-08-26T14:01:25.644 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:rbd: error opening image image_13: (2) No such file or directory 2023-08-26T14:01:25.648 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ current_image_id= 2023-08-26T14:01:25.648 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ test -n '' 2023-08-26T14:01:25.648 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ test deleted = deleted 2023-08-26T14:01:25.648 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ return 0 2023-08-26T14:01:25.649 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-08-26T14:01:25.649 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ image=image_14 2023-08-26T14:01:25.649 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ wait_for_image_present cluster1 mirror image_14 deleted 2023-08-26T14:01:25.649 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster1 2023-08-26T14:01:25.649 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T14:01:25.650 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=image_14 2023-08-26T14:01:25.650 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local state=deleted 2023-08-26T14:01:25.650 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image_id= 2023-08-26T14:01:25.650 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local s 2023-08-26T14:01:25.650 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ test -n '' 2023-08-26T14:01:25.650 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ get_image_id cluster1 mirror image_14 2023-08-26T14:01:25.651 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local cluster=cluster1 2023-08-26T14:01:25.651 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local pool=mirror 2023-08-26T14:01:25.651 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local image=image_14 2023-08-26T14:01:25.651 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ rbd --cluster=cluster1 info mirror/image_14 2023-08-26T14:01:25.651 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2023-08-26T14:01:25.679 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:rbd: error opening image image_14: (2) No such file or directory 2023-08-26T14:01:25.681 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ image_id= 2023-08-26T14:01:25.681 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2023-08-26T14:01:25.682 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ sleep 0.1 2023-08-26T14:01:25.784 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ test_image_present cluster1 mirror image_14 deleted '' 2023-08-26T14:01:25.785 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster1 2023-08-26T14:01:25.785 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T14:01:25.786 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=image_14 2023-08-26T14:01:25.786 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local test_state=deleted 2023-08-26T14:01:25.786 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image_id= 2023-08-26T14:01:25.786 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local current_state=deleted 2023-08-26T14:01:25.787 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local current_image_id 2023-08-26T14:01:25.787 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ get_image_id cluster1 mirror image_14 2023-08-26T14:01:25.787 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local cluster=cluster1 2023-08-26T14:01:25.788 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local pool=mirror 2023-08-26T14:01:25.788 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local image=image_14 2023-08-26T14:01:25.789 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ rbd --cluster=cluster1 info mirror/image_14 2023-08-26T14:01:25.789 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2023-08-26T14:01:25.814 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:rbd: error opening image image_14: (2) No such file or directory 2023-08-26T14:01:25.819 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ current_image_id= 2023-08-26T14:01:25.820 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ test -n '' 2023-08-26T14:01:25.820 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ test deleted = deleted 2023-08-26T14:01:25.820 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ return 0 2023-08-26T14:01:25.821 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-08-26T14:01:25.821 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ image=image_15 2023-08-26T14:01:25.821 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ wait_for_image_present cluster1 mirror image_15 deleted 2023-08-26T14:01:25.822 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster1 2023-08-26T14:01:25.822 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T14:01:25.823 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=image_15 2023-08-26T14:01:25.823 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local state=deleted 2023-08-26T14:01:25.823 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image_id= 2023-08-26T14:01:25.824 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local s 2023-08-26T14:01:25.824 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ test -n '' 2023-08-26T14:01:25.824 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ get_image_id cluster1 mirror image_15 2023-08-26T14:01:25.825 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local cluster=cluster1 2023-08-26T14:01:25.825 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local pool=mirror 2023-08-26T14:01:25.825 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local image=image_15 2023-08-26T14:01:25.826 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ rbd --cluster=cluster1 info mirror/image_15 2023-08-26T14:01:25.826 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2023-08-26T14:01:25.849 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:rbd: error opening image image_15: (2) No such file or directory 2023-08-26T14:01:25.852 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ image_id= 2023-08-26T14:01:25.852 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2023-08-26T14:01:25.853 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ sleep 0.1 2023-08-26T14:01:25.955 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ test_image_present cluster1 mirror image_15 deleted '' 2023-08-26T14:01:25.955 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster1 2023-08-26T14:01:25.956 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T14:01:25.956 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=image_15 2023-08-26T14:01:25.956 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local test_state=deleted 2023-08-26T14:01:25.957 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image_id= 2023-08-26T14:01:25.957 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local current_state=deleted 2023-08-26T14:01:25.957 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local current_image_id 2023-08-26T14:01:25.958 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ get_image_id cluster1 mirror image_15 2023-08-26T14:01:25.958 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local cluster=cluster1 2023-08-26T14:01:25.958 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local pool=mirror 2023-08-26T14:01:25.959 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local image=image_15 2023-08-26T14:01:25.959 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ rbd --cluster=cluster1 info mirror/image_15 2023-08-26T14:01:25.959 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2023-08-26T14:01:25.986 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:rbd: error opening image image_15: (2) No such file or directory 2023-08-26T14:01:25.990 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ current_image_id= 2023-08-26T14:01:25.990 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ test -n '' 2023-08-26T14:01:25.991 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ test deleted = deleted 2023-08-26T14:01:25.991 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ return 0 2023-08-26T14:01:25.991 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-08-26T14:01:25.991 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ image=image_16 2023-08-26T14:01:25.991 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ wait_for_image_present cluster1 mirror image_16 deleted 2023-08-26T14:01:25.992 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster1 2023-08-26T14:01:25.992 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T14:01:25.992 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=image_16 2023-08-26T14:01:25.992 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local state=deleted 2023-08-26T14:01:25.992 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image_id= 2023-08-26T14:01:25.993 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local s 2023-08-26T14:01:25.993 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ test -n '' 2023-08-26T14:01:25.993 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ get_image_id cluster1 mirror image_16 2023-08-26T14:01:25.993 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local cluster=cluster1 2023-08-26T14:01:25.993 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local pool=mirror 2023-08-26T14:01:25.994 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local image=image_16 2023-08-26T14:01:25.994 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ rbd --cluster=cluster1 info mirror/image_16 2023-08-26T14:01:25.994 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2023-08-26T14:01:26.020 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:rbd: error opening image image_16: (2) No such file or directory 2023-08-26T14:01:26.023 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ image_id= 2023-08-26T14:01:26.024 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2023-08-26T14:01:26.024 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ sleep 0.1 2023-08-26T14:01:26.126 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ test_image_present cluster1 mirror image_16 deleted '' 2023-08-26T14:01:26.126 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster1 2023-08-26T14:01:26.127 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T14:01:26.127 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=image_16 2023-08-26T14:01:26.127 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local test_state=deleted 2023-08-26T14:01:26.128 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image_id= 2023-08-26T14:01:26.128 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local current_state=deleted 2023-08-26T14:01:26.128 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local current_image_id 2023-08-26T14:01:26.128 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ get_image_id cluster1 mirror image_16 2023-08-26T14:01:26.129 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local cluster=cluster1 2023-08-26T14:01:26.129 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local pool=mirror 2023-08-26T14:01:26.129 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local image=image_16 2023-08-26T14:01:26.129 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ rbd --cluster=cluster1 info mirror/image_16 2023-08-26T14:01:26.129 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2023-08-26T14:01:26.156 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:rbd: error opening image image_16: (2) No such file or directory 2023-08-26T14:01:26.161 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ current_image_id= 2023-08-26T14:01:26.161 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ test -n '' 2023-08-26T14:01:26.161 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ test deleted = deleted 2023-08-26T14:01:26.162 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ return 0 2023-08-26T14:01:26.162 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-08-26T14:01:26.162 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ image=image_17 2023-08-26T14:01:26.162 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ wait_for_image_present cluster1 mirror image_17 deleted 2023-08-26T14:01:26.162 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster1 2023-08-26T14:01:26.163 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T14:01:26.163 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=image_17 2023-08-26T14:01:26.163 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local state=deleted 2023-08-26T14:01:26.163 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image_id= 2023-08-26T14:01:26.163 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local s 2023-08-26T14:01:26.164 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ test -n '' 2023-08-26T14:01:26.164 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ get_image_id cluster1 mirror image_17 2023-08-26T14:01:26.164 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local cluster=cluster1 2023-08-26T14:01:26.164 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local pool=mirror 2023-08-26T14:01:26.164 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local image=image_17 2023-08-26T14:01:26.164 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ rbd --cluster=cluster1 info mirror/image_17 2023-08-26T14:01:26.165 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2023-08-26T14:01:26.191 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:rbd: error opening image image_17: (2) No such file or directory 2023-08-26T14:01:26.194 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ image_id= 2023-08-26T14:01:26.194 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2023-08-26T14:01:26.194 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ sleep 0.1 2023-08-26T14:01:26.296 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ test_image_present cluster1 mirror image_17 deleted '' 2023-08-26T14:01:26.296 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster1 2023-08-26T14:01:26.296 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T14:01:26.296 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=image_17 2023-08-26T14:01:26.297 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local test_state=deleted 2023-08-26T14:01:26.297 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image_id= 2023-08-26T14:01:26.297 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local current_state=deleted 2023-08-26T14:01:26.297 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local current_image_id 2023-08-26T14:01:26.297 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ get_image_id cluster1 mirror image_17 2023-08-26T14:01:26.298 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local cluster=cluster1 2023-08-26T14:01:26.298 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local pool=mirror 2023-08-26T14:01:26.298 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local image=image_17 2023-08-26T14:01:26.298 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ rbd --cluster=cluster1 info mirror/image_17 2023-08-26T14:01:26.299 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2023-08-26T14:01:26.328 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:rbd: error opening image image_17: (2) No such file or directory 2023-08-26T14:01:26.332 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ current_image_id= 2023-08-26T14:01:26.333 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ test -n '' 2023-08-26T14:01:26.333 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ test deleted = deleted 2023-08-26T14:01:26.333 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ return 0 2023-08-26T14:01:26.333 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-08-26T14:01:26.333 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ image=image_18 2023-08-26T14:01:26.334 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ wait_for_image_present cluster1 mirror image_18 deleted 2023-08-26T14:01:26.334 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster1 2023-08-26T14:01:26.334 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T14:01:26.334 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=image_18 2023-08-26T14:01:26.334 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local state=deleted 2023-08-26T14:01:26.335 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image_id= 2023-08-26T14:01:26.335 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local s 2023-08-26T14:01:26.335 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ test -n '' 2023-08-26T14:01:26.335 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ get_image_id cluster1 mirror image_18 2023-08-26T14:01:26.336 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local cluster=cluster1 2023-08-26T14:01:26.336 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local pool=mirror 2023-08-26T14:01:26.336 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local image=image_18 2023-08-26T14:01:26.336 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ rbd --cluster=cluster1 info mirror/image_18 2023-08-26T14:01:26.336 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2023-08-26T14:01:26.364 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:rbd: error opening image image_18: (2) No such file or directory 2023-08-26T14:01:26.367 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ image_id= 2023-08-26T14:01:26.367 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2023-08-26T14:01:26.367 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ sleep 0.1 2023-08-26T14:01:26.469 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ test_image_present cluster1 mirror image_18 deleted '' 2023-08-26T14:01:26.469 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster1 2023-08-26T14:01:26.469 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T14:01:26.469 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=image_18 2023-08-26T14:01:26.470 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local test_state=deleted 2023-08-26T14:01:26.470 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image_id= 2023-08-26T14:01:26.470 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local current_state=deleted 2023-08-26T14:01:26.470 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local current_image_id 2023-08-26T14:01:26.470 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ get_image_id cluster1 mirror image_18 2023-08-26T14:01:26.470 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local cluster=cluster1 2023-08-26T14:01:26.471 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local pool=mirror 2023-08-26T14:01:26.471 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local image=image_18 2023-08-26T14:01:26.471 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ rbd --cluster=cluster1 info mirror/image_18 2023-08-26T14:01:26.471 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2023-08-26T14:01:26.501 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:rbd: error opening image image_18: (2) No such file or directory 2023-08-26T14:01:26.505 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ current_image_id= 2023-08-26T14:01:26.505 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ test -n '' 2023-08-26T14:01:26.505 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ test deleted = deleted 2023-08-26T14:01:26.506 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ return 0 2023-08-26T14:01:26.506 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-08-26T14:01:26.506 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ image=image_19 2023-08-26T14:01:26.506 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ wait_for_image_present cluster1 mirror image_19 deleted 2023-08-26T14:01:26.507 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster1 2023-08-26T14:01:26.507 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T14:01:26.507 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=image_19 2023-08-26T14:01:26.507 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local state=deleted 2023-08-26T14:01:26.508 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image_id= 2023-08-26T14:01:26.508 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local s 2023-08-26T14:01:26.508 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ test -n '' 2023-08-26T14:01:26.508 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ get_image_id cluster1 mirror image_19 2023-08-26T14:01:26.508 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local cluster=cluster1 2023-08-26T14:01:26.509 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local pool=mirror 2023-08-26T14:01:26.509 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local image=image_19 2023-08-26T14:01:26.509 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ rbd --cluster=cluster1 info mirror/image_19 2023-08-26T14:01:26.509 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2023-08-26T14:01:26.537 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:rbd: error opening image image_19: (2) No such file or directory 2023-08-26T14:01:26.539 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ image_id= 2023-08-26T14:01:26.540 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2023-08-26T14:01:26.540 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ sleep 0.1 2023-08-26T14:01:26.641 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ test_image_present cluster1 mirror image_19 deleted '' 2023-08-26T14:01:26.642 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster1 2023-08-26T14:01:26.642 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T14:01:26.642 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=image_19 2023-08-26T14:01:26.642 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local test_state=deleted 2023-08-26T14:01:26.643 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image_id= 2023-08-26T14:01:26.643 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local current_state=deleted 2023-08-26T14:01:26.643 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local current_image_id 2023-08-26T14:01:26.643 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ get_image_id cluster1 mirror image_19 2023-08-26T14:01:26.643 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local cluster=cluster1 2023-08-26T14:01:26.644 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local pool=mirror 2023-08-26T14:01:26.644 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local image=image_19 2023-08-26T14:01:26.644 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ rbd --cluster=cluster1 info mirror/image_19 2023-08-26T14:01:26.644 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2023-08-26T14:01:26.673 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:rbd: error opening image image_19: (2) No such file or directory 2023-08-26T14:01:26.677 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ current_image_id= 2023-08-26T14:01:26.677 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ test -n '' 2023-08-26T14:01:26.677 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ test deleted = deleted 2023-08-26T14:01:26.678 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ return 0 2023-08-26T14:01:26.678 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-08-26T14:01:26.678 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ image=image_20 2023-08-26T14:01:26.678 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ wait_for_image_present cluster1 mirror image_20 deleted 2023-08-26T14:01:26.678 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster1 2023-08-26T14:01:26.679 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T14:01:26.679 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=image_20 2023-08-26T14:01:26.679 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local state=deleted 2023-08-26T14:01:26.679 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image_id= 2023-08-26T14:01:26.679 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local s 2023-08-26T14:01:26.680 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ test -n '' 2023-08-26T14:01:26.680 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ get_image_id cluster1 mirror image_20 2023-08-26T14:01:26.680 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local cluster=cluster1 2023-08-26T14:01:26.680 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local pool=mirror 2023-08-26T14:01:26.680 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local image=image_20 2023-08-26T14:01:26.681 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ rbd --cluster=cluster1 info mirror/image_20 2023-08-26T14:01:26.681 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2023-08-26T14:01:26.708 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:rbd: error opening image image_20: (2) No such file or directory 2023-08-26T14:01:26.711 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ image_id= 2023-08-26T14:01:26.711 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2023-08-26T14:01:26.711 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ sleep 0.1 2023-08-26T14:01:26.813 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ test_image_present cluster1 mirror image_20 deleted '' 2023-08-26T14:01:26.813 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster1 2023-08-26T14:01:26.813 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T14:01:26.813 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=image_20 2023-08-26T14:01:26.814 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local test_state=deleted 2023-08-26T14:01:26.814 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image_id= 2023-08-26T14:01:26.814 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local current_state=deleted 2023-08-26T14:01:26.814 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local current_image_id 2023-08-26T14:01:26.814 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ get_image_id cluster1 mirror image_20 2023-08-26T14:01:26.815 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local cluster=cluster1 2023-08-26T14:01:26.815 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local pool=mirror 2023-08-26T14:01:26.815 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local image=image_20 2023-08-26T14:01:26.815 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ rbd --cluster=cluster1 info mirror/image_20 2023-08-26T14:01:26.815 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2023-08-26T14:01:26.844 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:rbd: error opening image image_20: (2) No such file or directory 2023-08-26T14:01:26.848 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ current_image_id= 2023-08-26T14:01:26.848 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ test -n '' 2023-08-26T14:01:26.848 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ test deleted = deleted 2023-08-26T14:01:26.848 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ return 0 2023-08-26T14:01:26.848 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-08-26T14:01:26.849 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ image=image_21 2023-08-26T14:01:26.849 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ wait_for_image_present cluster1 mirror image_21 deleted 2023-08-26T14:01:26.849 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster1 2023-08-26T14:01:26.849 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T14:01:26.849 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=image_21 2023-08-26T14:01:26.849 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local state=deleted 2023-08-26T14:01:26.850 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image_id= 2023-08-26T14:01:26.850 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local s 2023-08-26T14:01:26.850 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ test -n '' 2023-08-26T14:01:26.850 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ get_image_id cluster1 mirror image_21 2023-08-26T14:01:26.850 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local cluster=cluster1 2023-08-26T14:01:26.851 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local pool=mirror 2023-08-26T14:01:26.851 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local image=image_21 2023-08-26T14:01:26.851 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ rbd --cluster=cluster1 info mirror/image_21 2023-08-26T14:01:26.851 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2023-08-26T14:01:26.878 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:rbd: error opening image image_21: (2) No such file or directory 2023-08-26T14:01:26.881 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ image_id= 2023-08-26T14:01:26.881 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2023-08-26T14:01:26.882 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ sleep 0.1 2023-08-26T14:01:26.983 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ test_image_present cluster1 mirror image_21 deleted '' 2023-08-26T14:01:26.983 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster1 2023-08-26T14:01:26.984 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T14:01:26.984 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=image_21 2023-08-26T14:01:26.984 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local test_state=deleted 2023-08-26T14:01:26.984 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image_id= 2023-08-26T14:01:26.984 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local current_state=deleted 2023-08-26T14:01:26.985 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local current_image_id 2023-08-26T14:01:26.985 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ get_image_id cluster1 mirror image_21 2023-08-26T14:01:26.985 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local cluster=cluster1 2023-08-26T14:01:26.985 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local pool=mirror 2023-08-26T14:01:26.985 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local image=image_21 2023-08-26T14:01:26.986 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ rbd --cluster=cluster1 info mirror/image_21 2023-08-26T14:01:26.986 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2023-08-26T14:01:27.014 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:rbd: error opening image image_21: (2) No such file or directory 2023-08-26T14:01:27.018 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ current_image_id= 2023-08-26T14:01:27.018 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ test -n '' 2023-08-26T14:01:27.018 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ test deleted = deleted 2023-08-26T14:01:27.018 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ return 0 2023-08-26T14:01:27.019 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-08-26T14:01:27.019 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ image=image_22 2023-08-26T14:01:27.019 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ wait_for_image_present cluster1 mirror image_22 deleted 2023-08-26T14:01:27.019 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster1 2023-08-26T14:01:27.019 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T14:01:27.020 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=image_22 2023-08-26T14:01:27.020 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local state=deleted 2023-08-26T14:01:27.020 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image_id= 2023-08-26T14:01:27.020 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local s 2023-08-26T14:01:27.021 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ test -n '' 2023-08-26T14:01:27.021 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ get_image_id cluster1 mirror image_22 2023-08-26T14:01:27.021 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local cluster=cluster1 2023-08-26T14:01:27.021 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local pool=mirror 2023-08-26T14:01:27.021 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local image=image_22 2023-08-26T14:01:27.021 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ rbd --cluster=cluster1 info mirror/image_22 2023-08-26T14:01:27.022 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2023-08-26T14:01:27.049 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:rbd: error opening image image_22: (2) No such file or directory 2023-08-26T14:01:27.052 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ image_id= 2023-08-26T14:01:27.053 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2023-08-26T14:01:27.053 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ sleep 0.1 2023-08-26T14:01:27.154 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ test_image_present cluster1 mirror image_22 deleted '' 2023-08-26T14:01:27.154 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster1 2023-08-26T14:01:27.154 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T14:01:27.154 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=image_22 2023-08-26T14:01:27.155 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local test_state=deleted 2023-08-26T14:01:27.155 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image_id= 2023-08-26T14:01:27.155 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local current_state=deleted 2023-08-26T14:01:27.155 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local current_image_id 2023-08-26T14:01:27.155 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ get_image_id cluster1 mirror image_22 2023-08-26T14:01:27.156 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local cluster=cluster1 2023-08-26T14:01:27.156 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local pool=mirror 2023-08-26T14:01:27.156 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local image=image_22 2023-08-26T14:01:27.156 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ rbd --cluster=cluster1 info mirror/image_22 2023-08-26T14:01:27.156 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2023-08-26T14:01:27.185 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:rbd: error opening image image_22: (2) No such file or directory 2023-08-26T14:01:27.189 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ current_image_id= 2023-08-26T14:01:27.189 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ test -n '' 2023-08-26T14:01:27.189 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ test deleted = deleted 2023-08-26T14:01:27.190 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ return 0 2023-08-26T14:01:27.190 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-08-26T14:01:27.190 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ image=image_23 2023-08-26T14:01:27.190 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ wait_for_image_present cluster1 mirror image_23 deleted 2023-08-26T14:01:27.190 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster1 2023-08-26T14:01:27.191 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T14:01:27.191 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=image_23 2023-08-26T14:01:27.191 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local state=deleted 2023-08-26T14:01:27.191 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image_id= 2023-08-26T14:01:27.191 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local s 2023-08-26T14:01:27.192 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ test -n '' 2023-08-26T14:01:27.192 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ get_image_id cluster1 mirror image_23 2023-08-26T14:01:27.192 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local cluster=cluster1 2023-08-26T14:01:27.192 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local pool=mirror 2023-08-26T14:01:27.192 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local image=image_23 2023-08-26T14:01:27.193 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ rbd --cluster=cluster1 info mirror/image_23 2023-08-26T14:01:27.193 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2023-08-26T14:01:27.220 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:rbd: error opening image image_23: (2) No such file or directory 2023-08-26T14:01:27.223 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ image_id= 2023-08-26T14:01:27.224 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2023-08-26T14:01:27.224 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ sleep 0.1 2023-08-26T14:01:27.325 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ test_image_present cluster1 mirror image_23 deleted '' 2023-08-26T14:01:27.325 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster1 2023-08-26T14:01:27.325 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T14:01:27.325 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=image_23 2023-08-26T14:01:27.326 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local test_state=deleted 2023-08-26T14:01:27.326 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image_id= 2023-08-26T14:01:27.326 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local current_state=deleted 2023-08-26T14:01:27.326 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local current_image_id 2023-08-26T14:01:27.326 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ get_image_id cluster1 mirror image_23 2023-08-26T14:01:27.327 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local cluster=cluster1 2023-08-26T14:01:27.327 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local pool=mirror 2023-08-26T14:01:27.327 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local image=image_23 2023-08-26T14:01:27.327 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ rbd --cluster=cluster1 info mirror/image_23 2023-08-26T14:01:27.327 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2023-08-26T14:01:27.357 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:rbd: error opening image image_23: (2) No such file or directory 2023-08-26T14:01:27.361 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ current_image_id= 2023-08-26T14:01:27.361 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ test -n '' 2023-08-26T14:01:27.361 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ test deleted = deleted 2023-08-26T14:01:27.362 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ return 0 2023-08-26T14:01:27.362 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-08-26T14:01:27.362 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ image=image_24 2023-08-26T14:01:27.362 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ wait_for_image_present cluster1 mirror image_24 deleted 2023-08-26T14:01:27.362 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster1 2023-08-26T14:01:27.363 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T14:01:27.363 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=image_24 2023-08-26T14:01:27.363 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local state=deleted 2023-08-26T14:01:27.363 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image_id= 2023-08-26T14:01:27.363 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local s 2023-08-26T14:01:27.364 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ test -n '' 2023-08-26T14:01:27.364 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ get_image_id cluster1 mirror image_24 2023-08-26T14:01:27.364 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local cluster=cluster1 2023-08-26T14:01:27.364 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local pool=mirror 2023-08-26T14:01:27.364 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local image=image_24 2023-08-26T14:01:27.365 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ rbd --cluster=cluster1 info mirror/image_24 2023-08-26T14:01:27.365 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2023-08-26T14:01:27.593 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:rbd: error opening image image_24: (2) No such file or directory 2023-08-26T14:01:27.596 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ image_id= 2023-08-26T14:01:27.596 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2023-08-26T14:01:27.596 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ sleep 0.1 2023-08-26T14:01:27.697 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ test_image_present cluster1 mirror image_24 deleted '' 2023-08-26T14:01:27.698 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster1 2023-08-26T14:01:27.698 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T14:01:27.698 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=image_24 2023-08-26T14:01:27.698 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local test_state=deleted 2023-08-26T14:01:27.698 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image_id= 2023-08-26T14:01:27.699 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local current_state=deleted 2023-08-26T14:01:27.699 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local current_image_id 2023-08-26T14:01:27.699 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ get_image_id cluster1 mirror image_24 2023-08-26T14:01:27.699 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local cluster=cluster1 2023-08-26T14:01:27.700 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local pool=mirror 2023-08-26T14:01:27.700 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local image=image_24 2023-08-26T14:01:27.700 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ rbd --cluster=cluster1 info mirror/image_24 2023-08-26T14:01:27.700 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2023-08-26T14:01:27.729 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:rbd: error opening image image_24: (2) No such file or directory 2023-08-26T14:01:27.734 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ current_image_id= 2023-08-26T14:01:27.734 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ test -n '' 2023-08-26T14:01:27.735 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ test deleted = deleted 2023-08-26T14:01:27.735 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ return 0 2023-08-26T14:01:27.736 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-08-26T14:01:27.736 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ image=image_25 2023-08-26T14:01:27.736 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ wait_for_image_present cluster1 mirror image_25 deleted 2023-08-26T14:01:27.737 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster1 2023-08-26T14:01:27.737 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T14:01:27.737 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=image_25 2023-08-26T14:01:27.737 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local state=deleted 2023-08-26T14:01:27.738 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image_id= 2023-08-26T14:01:27.738 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local s 2023-08-26T14:01:27.738 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ test -n '' 2023-08-26T14:01:27.738 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ get_image_id cluster1 mirror image_25 2023-08-26T14:01:27.739 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local cluster=cluster1 2023-08-26T14:01:27.739 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local pool=mirror 2023-08-26T14:01:27.739 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local image=image_25 2023-08-26T14:01:27.739 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ rbd --cluster=cluster1 info mirror/image_25 2023-08-26T14:01:27.740 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2023-08-26T14:01:27.764 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:rbd: error opening image image_25: (2) No such file or directory 2023-08-26T14:01:27.767 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ image_id= 2023-08-26T14:01:27.767 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2023-08-26T14:01:27.768 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ sleep 0.1 2023-08-26T14:01:27.869 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ test_image_present cluster1 mirror image_25 deleted '' 2023-08-26T14:01:27.870 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster1 2023-08-26T14:01:27.870 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T14:01:27.870 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=image_25 2023-08-26T14:01:27.870 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local test_state=deleted 2023-08-26T14:01:27.870 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image_id= 2023-08-26T14:01:27.871 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local current_state=deleted 2023-08-26T14:01:27.871 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local current_image_id 2023-08-26T14:01:27.871 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ get_image_id cluster1 mirror image_25 2023-08-26T14:01:27.871 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local cluster=cluster1 2023-08-26T14:01:27.872 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local pool=mirror 2023-08-26T14:01:27.872 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local image=image_25 2023-08-26T14:01:27.872 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ rbd --cluster=cluster1 info mirror/image_25 2023-08-26T14:01:27.873 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2023-08-26T14:01:27.901 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:rbd: error opening image image_25: (2) No such file or directory 2023-08-26T14:01:27.905 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ current_image_id= 2023-08-26T14:01:27.905 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ test -n '' 2023-08-26T14:01:27.905 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ test deleted = deleted 2023-08-26T14:01:27.905 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ return 0 2023-08-26T14:01:27.906 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-08-26T14:01:27.906 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ image=image_26 2023-08-26T14:01:27.906 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ wait_for_image_present cluster1 mirror image_26 deleted 2023-08-26T14:01:27.906 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster1 2023-08-26T14:01:27.906 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T14:01:27.906 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=image_26 2023-08-26T14:01:27.907 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local state=deleted 2023-08-26T14:01:27.907 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image_id= 2023-08-26T14:01:27.907 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local s 2023-08-26T14:01:27.907 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ test -n '' 2023-08-26T14:01:27.907 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ get_image_id cluster1 mirror image_26 2023-08-26T14:01:27.908 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local cluster=cluster1 2023-08-26T14:01:27.908 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local pool=mirror 2023-08-26T14:01:27.908 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local image=image_26 2023-08-26T14:01:27.908 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ rbd --cluster=cluster1 info mirror/image_26 2023-08-26T14:01:27.908 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2023-08-26T14:01:27.936 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:rbd: error opening image image_26: (2) No such file or directory 2023-08-26T14:01:27.939 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ image_id= 2023-08-26T14:01:27.939 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2023-08-26T14:01:27.939 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ sleep 0.1 2023-08-26T14:01:28.040 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ test_image_present cluster1 mirror image_26 deleted '' 2023-08-26T14:01:28.041 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster1 2023-08-26T14:01:28.041 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T14:01:28.041 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=image_26 2023-08-26T14:01:28.041 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local test_state=deleted 2023-08-26T14:01:28.042 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image_id= 2023-08-26T14:01:28.042 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local current_state=deleted 2023-08-26T14:01:28.042 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local current_image_id 2023-08-26T14:01:28.042 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ get_image_id cluster1 mirror image_26 2023-08-26T14:01:28.042 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local cluster=cluster1 2023-08-26T14:01:28.043 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local pool=mirror 2023-08-26T14:01:28.043 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local image=image_26 2023-08-26T14:01:28.043 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ rbd --cluster=cluster1 info mirror/image_26 2023-08-26T14:01:28.043 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2023-08-26T14:01:28.071 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:rbd: error opening image image_26: (2) No such file or directory 2023-08-26T14:01:28.076 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ current_image_id= 2023-08-26T14:01:28.076 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ test -n '' 2023-08-26T14:01:28.076 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ test deleted = deleted 2023-08-26T14:01:28.076 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ return 0 2023-08-26T14:01:28.077 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-08-26T14:01:28.077 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ image=image_27 2023-08-26T14:01:28.077 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ wait_for_image_present cluster1 mirror image_27 deleted 2023-08-26T14:01:28.077 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster1 2023-08-26T14:01:28.077 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T14:01:28.077 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=image_27 2023-08-26T14:01:28.078 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local state=deleted 2023-08-26T14:01:28.078 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image_id= 2023-08-26T14:01:28.078 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local s 2023-08-26T14:01:28.078 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ test -n '' 2023-08-26T14:01:28.078 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ get_image_id cluster1 mirror image_27 2023-08-26T14:01:28.079 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local cluster=cluster1 2023-08-26T14:01:28.079 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local pool=mirror 2023-08-26T14:01:28.079 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local image=image_27 2023-08-26T14:01:28.079 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ rbd --cluster=cluster1 info mirror/image_27 2023-08-26T14:01:28.079 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2023-08-26T14:01:28.107 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:rbd: error opening image image_27: (2) No such file or directory 2023-08-26T14:01:28.110 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ image_id= 2023-08-26T14:01:28.110 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2023-08-26T14:01:28.110 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ sleep 0.1 2023-08-26T14:01:28.212 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ test_image_present cluster1 mirror image_27 deleted '' 2023-08-26T14:01:28.212 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster1 2023-08-26T14:01:28.212 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T14:01:28.212 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=image_27 2023-08-26T14:01:28.213 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local test_state=deleted 2023-08-26T14:01:28.213 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image_id= 2023-08-26T14:01:28.213 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local current_state=deleted 2023-08-26T14:01:28.213 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local current_image_id 2023-08-26T14:01:28.213 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ get_image_id cluster1 mirror image_27 2023-08-26T14:01:28.214 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local cluster=cluster1 2023-08-26T14:01:28.214 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local pool=mirror 2023-08-26T14:01:28.214 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local image=image_27 2023-08-26T14:01:28.214 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ rbd --cluster=cluster1 info mirror/image_27 2023-08-26T14:01:28.214 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2023-08-26T14:01:28.243 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:rbd: error opening image image_27: (2) No such file or directory 2023-08-26T14:01:28.247 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ current_image_id= 2023-08-26T14:01:28.247 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ test -n '' 2023-08-26T14:01:28.247 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ test deleted = deleted 2023-08-26T14:01:28.247 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ return 0 2023-08-26T14:01:28.248 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-08-26T14:01:28.248 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ image=image_28 2023-08-26T14:01:28.248 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ wait_for_image_present cluster1 mirror image_28 deleted 2023-08-26T14:01:28.248 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster1 2023-08-26T14:01:28.248 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T14:01:28.249 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=image_28 2023-08-26T14:01:28.249 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local state=deleted 2023-08-26T14:01:28.249 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image_id= 2023-08-26T14:01:28.249 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local s 2023-08-26T14:01:28.249 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ test -n '' 2023-08-26T14:01:28.250 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ get_image_id cluster1 mirror image_28 2023-08-26T14:01:28.250 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local cluster=cluster1 2023-08-26T14:01:28.250 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local pool=mirror 2023-08-26T14:01:28.250 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local image=image_28 2023-08-26T14:01:28.250 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ rbd --cluster=cluster1 info mirror/image_28 2023-08-26T14:01:28.250 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2023-08-26T14:01:28.279 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:rbd: error opening image image_28: (2) No such file or directory 2023-08-26T14:01:28.282 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ image_id= 2023-08-26T14:01:28.282 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2023-08-26T14:01:28.282 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ sleep 0.1 2023-08-26T14:01:28.383 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ test_image_present cluster1 mirror image_28 deleted '' 2023-08-26T14:01:28.384 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster1 2023-08-26T14:01:28.384 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T14:01:28.384 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=image_28 2023-08-26T14:01:28.384 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local test_state=deleted 2023-08-26T14:01:28.384 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image_id= 2023-08-26T14:01:28.385 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local current_state=deleted 2023-08-26T14:01:28.385 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local current_image_id 2023-08-26T14:01:28.385 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ get_image_id cluster1 mirror image_28 2023-08-26T14:01:28.385 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local cluster=cluster1 2023-08-26T14:01:28.385 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local pool=mirror 2023-08-26T14:01:28.386 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local image=image_28 2023-08-26T14:01:28.386 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ rbd --cluster=cluster1 info mirror/image_28 2023-08-26T14:01:28.386 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2023-08-26T14:01:28.415 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:rbd: error opening image image_28: (2) No such file or directory 2023-08-26T14:01:28.420 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ current_image_id= 2023-08-26T14:01:28.420 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ test -n '' 2023-08-26T14:01:28.421 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ test deleted = deleted 2023-08-26T14:01:28.421 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ return 0 2023-08-26T14:01:28.422 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-08-26T14:01:28.422 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ image=image_29 2023-08-26T14:01:28.422 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ wait_for_image_present cluster1 mirror image_29 deleted 2023-08-26T14:01:28.423 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster1 2023-08-26T14:01:28.423 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T14:01:28.423 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=image_29 2023-08-26T14:01:28.424 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local state=deleted 2023-08-26T14:01:28.424 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image_id= 2023-08-26T14:01:28.424 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local s 2023-08-26T14:01:28.425 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ test -n '' 2023-08-26T14:01:28.425 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ get_image_id cluster1 mirror image_29 2023-08-26T14:01:28.425 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local cluster=cluster1 2023-08-26T14:01:28.425 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local pool=mirror 2023-08-26T14:01:28.426 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local image=image_29 2023-08-26T14:01:28.426 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ rbd --cluster=cluster1 info mirror/image_29 2023-08-26T14:01:28.426 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2023-08-26T14:01:28.449 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:rbd: error opening image image_29: (2) No such file or directory 2023-08-26T14:01:28.452 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ image_id= 2023-08-26T14:01:28.452 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2023-08-26T14:01:28.453 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ sleep 0.1 2023-08-26T14:01:28.554 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ test_image_present cluster1 mirror image_29 deleted '' 2023-08-26T14:01:28.554 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster1 2023-08-26T14:01:28.555 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T14:01:28.555 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=image_29 2023-08-26T14:01:28.555 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local test_state=deleted 2023-08-26T14:01:28.555 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image_id= 2023-08-26T14:01:28.555 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local current_state=deleted 2023-08-26T14:01:28.556 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local current_image_id 2023-08-26T14:01:28.556 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ get_image_id cluster1 mirror image_29 2023-08-26T14:01:28.556 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local cluster=cluster1 2023-08-26T14:01:28.556 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local pool=mirror 2023-08-26T14:01:28.557 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local image=image_29 2023-08-26T14:01:28.557 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ rbd --cluster=cluster1 info mirror/image_29 2023-08-26T14:01:28.557 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2023-08-26T14:01:28.585 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:rbd: error opening image image_29: (2) No such file or directory 2023-08-26T14:01:28.589 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ current_image_id= 2023-08-26T14:01:28.589 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ test -n '' 2023-08-26T14:01:28.590 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ test deleted = deleted 2023-08-26T14:01:28.590 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ return 0 2023-08-26T14:01:28.590 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-08-26T14:01:28.590 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ image=image_30 2023-08-26T14:01:28.590 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ wait_for_image_present cluster1 mirror image_30 deleted 2023-08-26T14:01:28.591 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster1 2023-08-26T14:01:28.591 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T14:01:28.591 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=image_30 2023-08-26T14:01:28.591 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local state=deleted 2023-08-26T14:01:28.591 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image_id= 2023-08-26T14:01:28.592 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local s 2023-08-26T14:01:28.592 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ test -n '' 2023-08-26T14:01:28.592 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ get_image_id cluster1 mirror image_30 2023-08-26T14:01:28.592 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local cluster=cluster1 2023-08-26T14:01:28.593 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local pool=mirror 2023-08-26T14:01:28.593 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local image=image_30 2023-08-26T14:01:28.593 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ rbd --cluster=cluster1 info mirror/image_30 2023-08-26T14:01:28.593 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2023-08-26T14:01:28.621 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:rbd: error opening image image_30: (2) No such file or directory 2023-08-26T14:01:28.624 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ image_id= 2023-08-26T14:01:28.624 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2023-08-26T14:01:28.624 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ sleep 0.1 2023-08-26T14:01:28.726 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ test_image_present cluster1 mirror image_30 deleted '' 2023-08-26T14:01:28.726 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster1 2023-08-26T14:01:28.726 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T14:01:28.727 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=image_30 2023-08-26T14:01:28.727 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local test_state=deleted 2023-08-26T14:01:28.727 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image_id= 2023-08-26T14:01:28.727 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local current_state=deleted 2023-08-26T14:01:28.728 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local current_image_id 2023-08-26T14:01:28.728 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ get_image_id cluster1 mirror image_30 2023-08-26T14:01:28.728 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local cluster=cluster1 2023-08-26T14:01:28.728 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local pool=mirror 2023-08-26T14:01:28.728 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local image=image_30 2023-08-26T14:01:28.729 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ rbd --cluster=cluster1 info mirror/image_30 2023-08-26T14:01:28.729 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2023-08-26T14:01:28.757 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:rbd: error opening image image_30: (2) No such file or directory 2023-08-26T14:01:28.761 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ current_image_id= 2023-08-26T14:01:28.761 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ test -n '' 2023-08-26T14:01:28.761 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ test deleted = deleted 2023-08-26T14:01:28.762 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ return 0 2023-08-26T14:01:28.762 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-08-26T14:01:28.762 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ image=image_31 2023-08-26T14:01:28.762 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ wait_for_image_present cluster1 mirror image_31 deleted 2023-08-26T14:01:28.762 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster1 2023-08-26T14:01:28.763 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T14:01:28.763 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=image_31 2023-08-26T14:01:28.763 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local state=deleted 2023-08-26T14:01:28.763 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image_id= 2023-08-26T14:01:28.764 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local s 2023-08-26T14:01:28.764 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ test -n '' 2023-08-26T14:01:28.764 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ get_image_id cluster1 mirror image_31 2023-08-26T14:01:28.764 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local cluster=cluster1 2023-08-26T14:01:28.764 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local pool=mirror 2023-08-26T14:01:28.764 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local image=image_31 2023-08-26T14:01:28.765 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ rbd --cluster=cluster1 info mirror/image_31 2023-08-26T14:01:28.765 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2023-08-26T14:01:28.792 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:rbd: error opening image image_31: (2) No such file or directory 2023-08-26T14:01:28.795 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ image_id= 2023-08-26T14:01:28.795 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2023-08-26T14:01:28.795 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ sleep 0.1 2023-08-26T14:01:28.897 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ test_image_present cluster1 mirror image_31 deleted '' 2023-08-26T14:01:28.897 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster1 2023-08-26T14:01:28.897 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T14:01:28.898 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=image_31 2023-08-26T14:01:28.898 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local test_state=deleted 2023-08-26T14:01:28.898 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image_id= 2023-08-26T14:01:28.898 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local current_state=deleted 2023-08-26T14:01:28.898 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local current_image_id 2023-08-26T14:01:28.899 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ get_image_id cluster1 mirror image_31 2023-08-26T14:01:28.899 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local cluster=cluster1 2023-08-26T14:01:28.899 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local pool=mirror 2023-08-26T14:01:28.899 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local image=image_31 2023-08-26T14:01:28.899 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ rbd --cluster=cluster1 info mirror/image_31 2023-08-26T14:01:28.900 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2023-08-26T14:01:28.929 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:rbd: error opening image image_31: (2) No such file or directory 2023-08-26T14:01:28.933 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ current_image_id= 2023-08-26T14:01:28.933 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ test -n '' 2023-08-26T14:01:28.933 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ test deleted = deleted 2023-08-26T14:01:28.933 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ return 0 2023-08-26T14:01:28.934 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-08-26T14:01:28.934 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ image=image_32 2023-08-26T14:01:28.934 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ wait_for_image_present cluster1 mirror image_32 deleted 2023-08-26T14:01:28.934 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster1 2023-08-26T14:01:28.934 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T14:01:28.935 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=image_32 2023-08-26T14:01:28.935 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local state=deleted 2023-08-26T14:01:28.935 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image_id= 2023-08-26T14:01:28.935 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local s 2023-08-26T14:01:28.935 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ test -n '' 2023-08-26T14:01:28.935 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ get_image_id cluster1 mirror image_32 2023-08-26T14:01:28.936 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local cluster=cluster1 2023-08-26T14:01:28.936 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local pool=mirror 2023-08-26T14:01:28.936 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local image=image_32 2023-08-26T14:01:28.936 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ rbd --cluster=cluster1 info mirror/image_32 2023-08-26T14:01:28.936 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2023-08-26T14:01:28.964 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:rbd: error opening image image_32: (2) No such file or directory 2023-08-26T14:01:28.967 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ image_id= 2023-08-26T14:01:28.967 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2023-08-26T14:01:28.967 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ sleep 0.1 2023-08-26T14:01:29.069 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ test_image_present cluster1 mirror image_32 deleted '' 2023-08-26T14:01:29.069 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster1 2023-08-26T14:01:29.069 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T14:01:29.069 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=image_32 2023-08-26T14:01:29.070 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local test_state=deleted 2023-08-26T14:01:29.070 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image_id= 2023-08-26T14:01:29.070 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local current_state=deleted 2023-08-26T14:01:29.070 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local current_image_id 2023-08-26T14:01:29.070 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ get_image_id cluster1 mirror image_32 2023-08-26T14:01:29.070 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local cluster=cluster1 2023-08-26T14:01:29.071 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local pool=mirror 2023-08-26T14:01:29.071 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local image=image_32 2023-08-26T14:01:29.071 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ rbd --cluster=cluster1 info mirror/image_32 2023-08-26T14:01:29.071 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2023-08-26T14:01:29.100 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:rbd: error opening image image_32: (2) No such file or directory 2023-08-26T14:01:29.104 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ current_image_id= 2023-08-26T14:01:29.105 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ test -n '' 2023-08-26T14:01:29.105 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ test deleted = deleted 2023-08-26T14:01:29.105 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ return 0 2023-08-26T14:01:29.105 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-08-26T14:01:29.106 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ image=image_33 2023-08-26T14:01:29.106 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ wait_for_image_present cluster1 mirror image_33 deleted 2023-08-26T14:01:29.106 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster1 2023-08-26T14:01:29.106 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T14:01:29.106 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=image_33 2023-08-26T14:01:29.106 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local state=deleted 2023-08-26T14:01:29.107 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image_id= 2023-08-26T14:01:29.107 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local s 2023-08-26T14:01:29.107 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ test -n '' 2023-08-26T14:01:29.107 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ get_image_id cluster1 mirror image_33 2023-08-26T14:01:29.108 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local cluster=cluster1 2023-08-26T14:01:29.108 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local pool=mirror 2023-08-26T14:01:29.108 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local image=image_33 2023-08-26T14:01:29.108 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ rbd --cluster=cluster1 info mirror/image_33 2023-08-26T14:01:29.109 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2023-08-26T14:01:29.136 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:rbd: error opening image image_33: (2) No such file or directory 2023-08-26T14:01:29.138 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ image_id= 2023-08-26T14:01:29.139 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2023-08-26T14:01:29.139 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ sleep 0.1 2023-08-26T14:01:29.240 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ test_image_present cluster1 mirror image_33 deleted '' 2023-08-26T14:01:29.241 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster1 2023-08-26T14:01:29.241 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T14:01:29.241 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=image_33 2023-08-26T14:01:29.241 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local test_state=deleted 2023-08-26T14:01:29.242 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image_id= 2023-08-26T14:01:29.242 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local current_state=deleted 2023-08-26T14:01:29.242 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local current_image_id 2023-08-26T14:01:29.242 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ get_image_id cluster1 mirror image_33 2023-08-26T14:01:29.242 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local cluster=cluster1 2023-08-26T14:01:29.242 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local pool=mirror 2023-08-26T14:01:29.243 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local image=image_33 2023-08-26T14:01:29.243 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ rbd --cluster=cluster1 info mirror/image_33 2023-08-26T14:01:29.243 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2023-08-26T14:01:29.271 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:rbd: error opening image image_33: (2) No such file or directory 2023-08-26T14:01:29.275 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ current_image_id= 2023-08-26T14:01:29.275 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ test -n '' 2023-08-26T14:01:29.276 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ test deleted = deleted 2023-08-26T14:01:29.276 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ return 0 2023-08-26T14:01:29.276 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-08-26T14:01:29.276 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ image=image_34 2023-08-26T14:01:29.276 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ wait_for_image_present cluster1 mirror image_34 deleted 2023-08-26T14:01:29.277 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster1 2023-08-26T14:01:29.277 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T14:01:29.277 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=image_34 2023-08-26T14:01:29.277 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local state=deleted 2023-08-26T14:01:29.277 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image_id= 2023-08-26T14:01:29.278 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local s 2023-08-26T14:01:29.278 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ test -n '' 2023-08-26T14:01:29.278 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ get_image_id cluster1 mirror image_34 2023-08-26T14:01:29.278 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local cluster=cluster1 2023-08-26T14:01:29.278 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local pool=mirror 2023-08-26T14:01:29.279 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local image=image_34 2023-08-26T14:01:29.279 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ rbd --cluster=cluster1 info mirror/image_34 2023-08-26T14:01:29.279 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2023-08-26T14:01:29.306 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:rbd: error opening image image_34: (2) No such file or directory 2023-08-26T14:01:29.309 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ image_id= 2023-08-26T14:01:29.309 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2023-08-26T14:01:29.310 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ sleep 0.1 2023-08-26T14:01:29.411 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ test_image_present cluster1 mirror image_34 deleted '' 2023-08-26T14:01:29.411 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster1 2023-08-26T14:01:29.412 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T14:01:29.412 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=image_34 2023-08-26T14:01:29.412 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local test_state=deleted 2023-08-26T14:01:29.412 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image_id= 2023-08-26T14:01:29.412 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local current_state=deleted 2023-08-26T14:01:29.413 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local current_image_id 2023-08-26T14:01:29.413 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ get_image_id cluster1 mirror image_34 2023-08-26T14:01:29.413 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local cluster=cluster1 2023-08-26T14:01:29.413 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local pool=mirror 2023-08-26T14:01:29.414 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local image=image_34 2023-08-26T14:01:29.414 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ rbd --cluster=cluster1 info mirror/image_34 2023-08-26T14:01:29.414 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2023-08-26T14:01:29.444 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:rbd: error opening image image_34: (2) No such file or directory 2023-08-26T14:01:29.448 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ current_image_id= 2023-08-26T14:01:29.449 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ test -n '' 2023-08-26T14:01:29.449 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ test deleted = deleted 2023-08-26T14:01:29.449 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ return 0 2023-08-26T14:01:29.450 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-08-26T14:01:29.450 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ image=image_35 2023-08-26T14:01:29.451 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ wait_for_image_present cluster1 mirror image_35 deleted 2023-08-26T14:01:29.451 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster1 2023-08-26T14:01:29.451 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T14:01:29.452 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=image_35 2023-08-26T14:01:29.452 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local state=deleted 2023-08-26T14:01:29.452 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image_id= 2023-08-26T14:01:29.452 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local s 2023-08-26T14:01:29.453 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ test -n '' 2023-08-26T14:01:29.453 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ get_image_id cluster1 mirror image_35 2023-08-26T14:01:29.453 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local cluster=cluster1 2023-08-26T14:01:29.454 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local pool=mirror 2023-08-26T14:01:29.454 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local image=image_35 2023-08-26T14:01:29.454 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ rbd --cluster=cluster1 info mirror/image_35 2023-08-26T14:01:29.455 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2023-08-26T14:01:29.480 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:rbd: error opening image image_35: (2) No such file or directory 2023-08-26T14:01:29.483 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ image_id= 2023-08-26T14:01:29.483 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2023-08-26T14:01:29.483 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ sleep 0.1 2023-08-26T14:01:29.584 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ test_image_present cluster1 mirror image_35 deleted '' 2023-08-26T14:01:29.585 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster1 2023-08-26T14:01:29.585 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T14:01:29.585 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=image_35 2023-08-26T14:01:29.585 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local test_state=deleted 2023-08-26T14:01:29.585 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image_id= 2023-08-26T14:01:29.586 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local current_state=deleted 2023-08-26T14:01:29.586 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local current_image_id 2023-08-26T14:01:29.586 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ get_image_id cluster1 mirror image_35 2023-08-26T14:01:29.586 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local cluster=cluster1 2023-08-26T14:01:29.586 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local pool=mirror 2023-08-26T14:01:29.586 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local image=image_35 2023-08-26T14:01:29.587 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ rbd --cluster=cluster1 info mirror/image_35 2023-08-26T14:01:29.587 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2023-08-26T14:01:29.616 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:rbd: error opening image image_35: (2) No such file or directory 2023-08-26T14:01:29.620 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ current_image_id= 2023-08-26T14:01:29.620 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ test -n '' 2023-08-26T14:01:29.620 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ test deleted = deleted 2023-08-26T14:01:29.620 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ return 0 2023-08-26T14:01:29.620 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-08-26T14:01:29.621 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ image=image_36 2023-08-26T14:01:29.621 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ wait_for_image_present cluster1 mirror image_36 deleted 2023-08-26T14:01:29.621 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster1 2023-08-26T14:01:29.621 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T14:01:29.621 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=image_36 2023-08-26T14:01:29.621 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local state=deleted 2023-08-26T14:01:29.622 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image_id= 2023-08-26T14:01:29.622 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local s 2023-08-26T14:01:29.622 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ test -n '' 2023-08-26T14:01:29.622 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ get_image_id cluster1 mirror image_36 2023-08-26T14:01:29.622 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local cluster=cluster1 2023-08-26T14:01:29.623 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local pool=mirror 2023-08-26T14:01:29.623 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local image=image_36 2023-08-26T14:01:29.623 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ rbd --cluster=cluster1 info mirror/image_36 2023-08-26T14:01:29.623 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2023-08-26T14:01:29.651 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:rbd: error opening image image_36: (2) No such file or directory 2023-08-26T14:01:29.654 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ image_id= 2023-08-26T14:01:29.654 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2023-08-26T14:01:29.654 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ sleep 0.1 2023-08-26T14:01:29.756 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ test_image_present cluster1 mirror image_36 deleted '' 2023-08-26T14:01:29.756 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster1 2023-08-26T14:01:29.756 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T14:01:29.757 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=image_36 2023-08-26T14:01:29.757 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local test_state=deleted 2023-08-26T14:01:29.757 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image_id= 2023-08-26T14:01:29.757 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local current_state=deleted 2023-08-26T14:01:29.757 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local current_image_id 2023-08-26T14:01:29.758 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ get_image_id cluster1 mirror image_36 2023-08-26T14:01:29.758 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local cluster=cluster1 2023-08-26T14:01:29.758 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local pool=mirror 2023-08-26T14:01:29.758 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local image=image_36 2023-08-26T14:01:29.758 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ rbd --cluster=cluster1 info mirror/image_36 2023-08-26T14:01:29.759 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2023-08-26T14:01:29.788 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:rbd: error opening image image_36: (2) No such file or directory 2023-08-26T14:01:29.792 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ current_image_id= 2023-08-26T14:01:29.792 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ test -n '' 2023-08-26T14:01:29.792 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ test deleted = deleted 2023-08-26T14:01:29.793 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ return 0 2023-08-26T14:01:29.793 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-08-26T14:01:29.793 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ image=image_37 2023-08-26T14:01:29.793 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ wait_for_image_present cluster1 mirror image_37 deleted 2023-08-26T14:01:29.793 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster1 2023-08-26T14:01:29.794 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T14:01:29.794 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=image_37 2023-08-26T14:01:29.794 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local state=deleted 2023-08-26T14:01:29.794 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image_id= 2023-08-26T14:01:29.794 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local s 2023-08-26T14:01:29.794 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ test -n '' 2023-08-26T14:01:29.795 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ get_image_id cluster1 mirror image_37 2023-08-26T14:01:29.795 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local cluster=cluster1 2023-08-26T14:01:29.795 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local pool=mirror 2023-08-26T14:01:29.795 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local image=image_37 2023-08-26T14:01:29.795 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ rbd --cluster=cluster1 info mirror/image_37 2023-08-26T14:01:29.796 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2023-08-26T14:01:29.824 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:rbd: error opening image image_37: (2) No such file or directory 2023-08-26T14:01:29.826 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ image_id= 2023-08-26T14:01:29.826 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2023-08-26T14:01:29.827 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ sleep 0.1 2023-08-26T14:01:29.928 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ test_image_present cluster1 mirror image_37 deleted '' 2023-08-26T14:01:29.928 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster1 2023-08-26T14:01:29.928 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T14:01:29.928 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=image_37 2023-08-26T14:01:29.929 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local test_state=deleted 2023-08-26T14:01:29.929 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image_id= 2023-08-26T14:01:29.929 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local current_state=deleted 2023-08-26T14:01:29.929 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local current_image_id 2023-08-26T14:01:29.929 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ get_image_id cluster1 mirror image_37 2023-08-26T14:01:29.930 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local cluster=cluster1 2023-08-26T14:01:29.930 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local pool=mirror 2023-08-26T14:01:29.930 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local image=image_37 2023-08-26T14:01:29.930 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ rbd --cluster=cluster1 info mirror/image_37 2023-08-26T14:01:29.930 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2023-08-26T14:01:29.960 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:rbd: error opening image image_37: (2) No such file or directory 2023-08-26T14:01:29.964 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ current_image_id= 2023-08-26T14:01:29.965 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ test -n '' 2023-08-26T14:01:29.965 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ test deleted = deleted 2023-08-26T14:01:29.965 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ return 0 2023-08-26T14:01:29.965 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-08-26T14:01:29.965 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ image=image_38 2023-08-26T14:01:29.966 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ wait_for_image_present cluster1 mirror image_38 deleted 2023-08-26T14:01:29.966 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster1 2023-08-26T14:01:29.966 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T14:01:29.966 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=image_38 2023-08-26T14:01:29.966 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local state=deleted 2023-08-26T14:01:29.967 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image_id= 2023-08-26T14:01:29.967 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local s 2023-08-26T14:01:29.967 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ test -n '' 2023-08-26T14:01:29.967 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ get_image_id cluster1 mirror image_38 2023-08-26T14:01:29.967 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local cluster=cluster1 2023-08-26T14:01:29.968 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local pool=mirror 2023-08-26T14:01:29.968 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local image=image_38 2023-08-26T14:01:29.968 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ rbd --cluster=cluster1 info mirror/image_38 2023-08-26T14:01:29.968 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2023-08-26T14:01:29.997 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:rbd: error opening image image_38: (2) No such file or directory 2023-08-26T14:01:29.999 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ image_id= 2023-08-26T14:01:30.000 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2023-08-26T14:01:30.000 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ sleep 0.1 2023-08-26T14:01:30.102 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ test_image_present cluster1 mirror image_38 deleted '' 2023-08-26T14:01:30.102 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster1 2023-08-26T14:01:30.102 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T14:01:30.102 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=image_38 2023-08-26T14:01:30.103 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local test_state=deleted 2023-08-26T14:01:30.103 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image_id= 2023-08-26T14:01:30.103 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local current_state=deleted 2023-08-26T14:01:30.103 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local current_image_id 2023-08-26T14:01:30.103 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ get_image_id cluster1 mirror image_38 2023-08-26T14:01:30.104 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local cluster=cluster1 2023-08-26T14:01:30.104 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local pool=mirror 2023-08-26T14:01:30.104 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local image=image_38 2023-08-26T14:01:30.104 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ rbd --cluster=cluster1 info mirror/image_38 2023-08-26T14:01:30.104 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2023-08-26T14:01:30.135 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:rbd: error opening image image_38: (2) No such file or directory 2023-08-26T14:01:30.139 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ current_image_id= 2023-08-26T14:01:30.139 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ test -n '' 2023-08-26T14:01:30.140 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ test deleted = deleted 2023-08-26T14:01:30.140 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ return 0 2023-08-26T14:01:30.140 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-08-26T14:01:30.140 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ image=image_39 2023-08-26T14:01:30.140 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ wait_for_image_present cluster1 mirror image_39 deleted 2023-08-26T14:01:30.141 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster1 2023-08-26T14:01:30.141 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T14:01:30.141 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=image_39 2023-08-26T14:01:30.141 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local state=deleted 2023-08-26T14:01:30.141 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image_id= 2023-08-26T14:01:30.141 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local s 2023-08-26T14:01:30.142 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ test -n '' 2023-08-26T14:01:30.142 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ get_image_id cluster1 mirror image_39 2023-08-26T14:01:30.142 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local cluster=cluster1 2023-08-26T14:01:30.142 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local pool=mirror 2023-08-26T14:01:30.142 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local image=image_39 2023-08-26T14:01:30.142 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ rbd --cluster=cluster1 info mirror/image_39 2023-08-26T14:01:30.143 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2023-08-26T14:01:30.171 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:rbd: error opening image image_39: (2) No such file or directory 2023-08-26T14:01:30.174 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ image_id= 2023-08-26T14:01:30.174 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2023-08-26T14:01:30.174 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ sleep 0.1 2023-08-26T14:01:30.275 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ test_image_present cluster1 mirror image_39 deleted '' 2023-08-26T14:01:30.276 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster1 2023-08-26T14:01:30.276 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T14:01:30.276 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=image_39 2023-08-26T14:01:30.276 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local test_state=deleted 2023-08-26T14:01:30.277 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image_id= 2023-08-26T14:01:30.277 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local current_state=deleted 2023-08-26T14:01:30.277 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local current_image_id 2023-08-26T14:01:30.277 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ get_image_id cluster1 mirror image_39 2023-08-26T14:01:30.277 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local cluster=cluster1 2023-08-26T14:01:30.278 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local pool=mirror 2023-08-26T14:01:30.278 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local image=image_39 2023-08-26T14:01:30.278 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ rbd --cluster=cluster1 info mirror/image_39 2023-08-26T14:01:30.278 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2023-08-26T14:01:30.309 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:rbd: error opening image image_39: (2) No such file or directory 2023-08-26T14:01:30.313 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ current_image_id= 2023-08-26T14:01:30.313 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ test -n '' 2023-08-26T14:01:30.313 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ test deleted = deleted 2023-08-26T14:01:30.313 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ return 0 2023-08-26T14:01:30.314 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-08-26T14:01:30.314 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ image=image_40 2023-08-26T14:01:30.314 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ wait_for_image_present cluster1 mirror image_40 deleted 2023-08-26T14:01:30.314 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster1 2023-08-26T14:01:30.314 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T14:01:30.315 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=image_40 2023-08-26T14:01:30.315 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local state=deleted 2023-08-26T14:01:30.315 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image_id= 2023-08-26T14:01:30.315 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local s 2023-08-26T14:01:30.315 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ test -n '' 2023-08-26T14:01:30.316 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ get_image_id cluster1 mirror image_40 2023-08-26T14:01:30.316 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local cluster=cluster1 2023-08-26T14:01:30.316 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local pool=mirror 2023-08-26T14:01:30.316 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local image=image_40 2023-08-26T14:01:30.316 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ rbd --cluster=cluster1 info mirror/image_40 2023-08-26T14:01:30.317 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2023-08-26T14:01:30.360 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:rbd: error opening image image_40: (2) No such file or directory 2023-08-26T14:01:30.363 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ image_id= 2023-08-26T14:01:30.363 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2023-08-26T14:01:30.364 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ sleep 0.1 2023-08-26T14:01:30.465 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ test_image_present cluster1 mirror image_40 deleted '' 2023-08-26T14:01:30.465 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster1 2023-08-26T14:01:30.465 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T14:01:30.466 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=image_40 2023-08-26T14:01:30.466 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local test_state=deleted 2023-08-26T14:01:30.466 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image_id= 2023-08-26T14:01:30.466 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local current_state=deleted 2023-08-26T14:01:30.466 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local current_image_id 2023-08-26T14:01:30.467 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ get_image_id cluster1 mirror image_40 2023-08-26T14:01:30.467 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local cluster=cluster1 2023-08-26T14:01:30.467 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local pool=mirror 2023-08-26T14:01:30.467 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local image=image_40 2023-08-26T14:01:30.467 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ rbd --cluster=cluster1 info mirror/image_40 2023-08-26T14:01:30.467 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2023-08-26T14:01:30.497 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:rbd: error opening image image_40: (2) No such file or directory 2023-08-26T14:01:30.501 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ current_image_id= 2023-08-26T14:01:30.501 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ test -n '' 2023-08-26T14:01:30.501 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ test deleted = deleted 2023-08-26T14:01:30.501 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ return 0 2023-08-26T14:01:30.502 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-08-26T14:01:30.502 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ image=image_41 2023-08-26T14:01:30.502 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ wait_for_image_present cluster1 mirror image_41 deleted 2023-08-26T14:01:30.502 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster1 2023-08-26T14:01:30.502 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T14:01:30.502 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=image_41 2023-08-26T14:01:30.503 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local state=deleted 2023-08-26T14:01:30.503 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image_id= 2023-08-26T14:01:30.503 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local s 2023-08-26T14:01:30.503 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ test -n '' 2023-08-26T14:01:30.503 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ get_image_id cluster1 mirror image_41 2023-08-26T14:01:30.504 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local cluster=cluster1 2023-08-26T14:01:30.504 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local pool=mirror 2023-08-26T14:01:30.504 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local image=image_41 2023-08-26T14:01:30.504 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ rbd --cluster=cluster1 info mirror/image_41 2023-08-26T14:01:30.504 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2023-08-26T14:01:30.532 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:rbd: error opening image image_41: (2) No such file or directory 2023-08-26T14:01:30.535 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ image_id= 2023-08-26T14:01:30.535 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2023-08-26T14:01:30.535 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ sleep 0.1 2023-08-26T14:01:30.637 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ test_image_present cluster1 mirror image_41 deleted '' 2023-08-26T14:01:30.637 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster1 2023-08-26T14:01:30.638 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T14:01:30.638 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=image_41 2023-08-26T14:01:30.638 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local test_state=deleted 2023-08-26T14:01:30.638 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image_id= 2023-08-26T14:01:30.638 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local current_state=deleted 2023-08-26T14:01:30.638 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local current_image_id 2023-08-26T14:01:30.639 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ get_image_id cluster1 mirror image_41 2023-08-26T14:01:30.639 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local cluster=cluster1 2023-08-26T14:01:30.639 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local pool=mirror 2023-08-26T14:01:30.639 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local image=image_41 2023-08-26T14:01:30.639 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ rbd --cluster=cluster1 info mirror/image_41 2023-08-26T14:01:30.640 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2023-08-26T14:01:30.669 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:rbd: error opening image image_41: (2) No such file or directory 2023-08-26T14:01:30.673 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ current_image_id= 2023-08-26T14:01:30.673 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ test -n '' 2023-08-26T14:01:30.674 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ test deleted = deleted 2023-08-26T14:01:30.674 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ return 0 2023-08-26T14:01:30.674 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-08-26T14:01:30.674 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ image=image_42 2023-08-26T14:01:30.674 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ wait_for_image_present cluster1 mirror image_42 deleted 2023-08-26T14:01:30.675 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster1 2023-08-26T14:01:30.675 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T14:01:30.675 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=image_42 2023-08-26T14:01:30.675 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local state=deleted 2023-08-26T14:01:30.675 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image_id= 2023-08-26T14:01:30.676 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local s 2023-08-26T14:01:30.676 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ test -n '' 2023-08-26T14:01:30.676 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ get_image_id cluster1 mirror image_42 2023-08-26T14:01:30.676 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local cluster=cluster1 2023-08-26T14:01:30.676 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local pool=mirror 2023-08-26T14:01:30.676 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local image=image_42 2023-08-26T14:01:30.677 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ rbd --cluster=cluster1 info mirror/image_42 2023-08-26T14:01:30.677 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2023-08-26T14:01:30.704 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:rbd: error opening image image_42: (2) No such file or directory 2023-08-26T14:01:30.708 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ image_id= 2023-08-26T14:01:30.708 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2023-08-26T14:01:30.708 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ sleep 0.1 2023-08-26T14:01:30.810 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ test_image_present cluster1 mirror image_42 deleted '' 2023-08-26T14:01:30.810 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster1 2023-08-26T14:01:30.811 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T14:01:30.811 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=image_42 2023-08-26T14:01:30.811 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local test_state=deleted 2023-08-26T14:01:30.811 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image_id= 2023-08-26T14:01:30.812 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local current_state=deleted 2023-08-26T14:01:30.812 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local current_image_id 2023-08-26T14:01:30.812 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ get_image_id cluster1 mirror image_42 2023-08-26T14:01:30.812 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local cluster=cluster1 2023-08-26T14:01:30.812 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local pool=mirror 2023-08-26T14:01:30.813 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local image=image_42 2023-08-26T14:01:30.813 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ rbd --cluster=cluster1 info mirror/image_42 2023-08-26T14:01:30.813 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2023-08-26T14:01:30.841 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:rbd: error opening image image_42: (2) No such file or directory 2023-08-26T14:01:30.845 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ current_image_id= 2023-08-26T14:01:30.846 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ test -n '' 2023-08-26T14:01:30.846 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ test deleted = deleted 2023-08-26T14:01:30.846 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ return 0 2023-08-26T14:01:30.846 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-08-26T14:01:30.847 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ image=image_43 2023-08-26T14:01:30.847 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ wait_for_image_present cluster1 mirror image_43 deleted 2023-08-26T14:01:30.847 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster1 2023-08-26T14:01:30.847 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T14:01:30.847 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=image_43 2023-08-26T14:01:30.848 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local state=deleted 2023-08-26T14:01:30.848 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image_id= 2023-08-26T14:01:30.848 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local s 2023-08-26T14:01:30.848 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ test -n '' 2023-08-26T14:01:30.848 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ get_image_id cluster1 mirror image_43 2023-08-26T14:01:30.849 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local cluster=cluster1 2023-08-26T14:01:30.849 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local pool=mirror 2023-08-26T14:01:30.849 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local image=image_43 2023-08-26T14:01:30.849 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ rbd --cluster=cluster1 info mirror/image_43 2023-08-26T14:01:30.849 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2023-08-26T14:01:30.877 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:rbd: error opening image image_43: (2) No such file or directory 2023-08-26T14:01:30.880 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ image_id= 2023-08-26T14:01:30.881 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2023-08-26T14:01:30.881 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ sleep 0.1 2023-08-26T14:01:30.982 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ test_image_present cluster1 mirror image_43 deleted '' 2023-08-26T14:01:30.983 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster1 2023-08-26T14:01:30.983 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T14:01:30.983 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=image_43 2023-08-26T14:01:30.983 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local test_state=deleted 2023-08-26T14:01:30.983 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image_id= 2023-08-26T14:01:30.984 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local current_state=deleted 2023-08-26T14:01:30.984 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local current_image_id 2023-08-26T14:01:30.984 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ get_image_id cluster1 mirror image_43 2023-08-26T14:01:30.984 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local cluster=cluster1 2023-08-26T14:01:30.984 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local pool=mirror 2023-08-26T14:01:30.984 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local image=image_43 2023-08-26T14:01:30.985 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ rbd --cluster=cluster1 info mirror/image_43 2023-08-26T14:01:30.985 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2023-08-26T14:01:31.014 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:rbd: error opening image image_43: (2) No such file or directory 2023-08-26T14:01:31.018 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ current_image_id= 2023-08-26T14:01:31.018 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ test -n '' 2023-08-26T14:01:31.018 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ test deleted = deleted 2023-08-26T14:01:31.019 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ return 0 2023-08-26T14:01:31.019 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-08-26T14:01:31.019 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ image=image_44 2023-08-26T14:01:31.019 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ wait_for_image_present cluster1 mirror image_44 deleted 2023-08-26T14:01:31.019 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster1 2023-08-26T14:01:31.019 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T14:01:31.020 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=image_44 2023-08-26T14:01:31.020 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local state=deleted 2023-08-26T14:01:31.020 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image_id= 2023-08-26T14:01:31.020 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local s 2023-08-26T14:01:31.020 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ test -n '' 2023-08-26T14:01:31.021 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ get_image_id cluster1 mirror image_44 2023-08-26T14:01:31.021 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local cluster=cluster1 2023-08-26T14:01:31.021 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local pool=mirror 2023-08-26T14:01:31.021 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local image=image_44 2023-08-26T14:01:31.021 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ rbd --cluster=cluster1 info mirror/image_44 2023-08-26T14:01:31.021 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2023-08-26T14:01:31.250 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:rbd: error opening image image_44: (2) No such file or directory 2023-08-26T14:01:31.253 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ image_id= 2023-08-26T14:01:31.253 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2023-08-26T14:01:31.253 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ sleep 0.1 2023-08-26T14:01:31.355 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ test_image_present cluster1 mirror image_44 deleted '' 2023-08-26T14:01:31.355 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster1 2023-08-26T14:01:31.355 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T14:01:31.355 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=image_44 2023-08-26T14:01:31.356 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local test_state=deleted 2023-08-26T14:01:31.356 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image_id= 2023-08-26T14:01:31.356 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local current_state=deleted 2023-08-26T14:01:31.356 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local current_image_id 2023-08-26T14:01:31.356 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ get_image_id cluster1 mirror image_44 2023-08-26T14:01:31.357 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local cluster=cluster1 2023-08-26T14:01:31.357 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local pool=mirror 2023-08-26T14:01:31.357 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local image=image_44 2023-08-26T14:01:31.357 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ rbd --cluster=cluster1 info mirror/image_44 2023-08-26T14:01:31.357 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2023-08-26T14:01:31.388 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:rbd: error opening image image_44: (2) No such file or directory 2023-08-26T14:01:31.392 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ current_image_id= 2023-08-26T14:01:31.392 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ test -n '' 2023-08-26T14:01:31.392 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ test deleted = deleted 2023-08-26T14:01:31.393 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ return 0 2023-08-26T14:01:31.393 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-08-26T14:01:31.393 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ image=image_45 2023-08-26T14:01:31.393 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ wait_for_image_present cluster1 mirror image_45 deleted 2023-08-26T14:01:31.393 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster1 2023-08-26T14:01:31.394 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T14:01:31.394 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=image_45 2023-08-26T14:01:31.394 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local state=deleted 2023-08-26T14:01:31.394 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image_id= 2023-08-26T14:01:31.394 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local s 2023-08-26T14:01:31.394 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ test -n '' 2023-08-26T14:01:31.395 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ get_image_id cluster1 mirror image_45 2023-08-26T14:01:31.395 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local cluster=cluster1 2023-08-26T14:01:31.395 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local pool=mirror 2023-08-26T14:01:31.395 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local image=image_45 2023-08-26T14:01:31.395 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ rbd --cluster=cluster1 info mirror/image_45 2023-08-26T14:01:31.396 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2023-08-26T14:01:31.424 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:rbd: error opening image image_45: (2) No such file or directory 2023-08-26T14:01:31.426 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ image_id= 2023-08-26T14:01:31.427 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2023-08-26T14:01:31.427 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ sleep 0.1 2023-08-26T14:01:31.529 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ test_image_present cluster1 mirror image_45 deleted '' 2023-08-26T14:01:31.529 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster1 2023-08-26T14:01:31.529 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T14:01:31.530 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=image_45 2023-08-26T14:01:31.530 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local test_state=deleted 2023-08-26T14:01:31.530 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image_id= 2023-08-26T14:01:31.530 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local current_state=deleted 2023-08-26T14:01:31.531 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local current_image_id 2023-08-26T14:01:31.531 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ get_image_id cluster1 mirror image_45 2023-08-26T14:01:31.531 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local cluster=cluster1 2023-08-26T14:01:31.531 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local pool=mirror 2023-08-26T14:01:31.532 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local image=image_45 2023-08-26T14:01:31.532 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ rbd --cluster=cluster1 info mirror/image_45 2023-08-26T14:01:31.532 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2023-08-26T14:01:31.560 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:rbd: error opening image image_45: (2) No such file or directory 2023-08-26T14:01:31.564 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ current_image_id= 2023-08-26T14:01:31.564 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ test -n '' 2023-08-26T14:01:31.564 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ test deleted = deleted 2023-08-26T14:01:31.564 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ return 0 2023-08-26T14:01:31.565 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-08-26T14:01:31.565 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ image=image_46 2023-08-26T14:01:31.565 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ wait_for_image_present cluster1 mirror image_46 deleted 2023-08-26T14:01:31.565 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster1 2023-08-26T14:01:31.565 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T14:01:31.566 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=image_46 2023-08-26T14:01:31.566 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local state=deleted 2023-08-26T14:01:31.566 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image_id= 2023-08-26T14:01:31.566 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local s 2023-08-26T14:01:31.566 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ test -n '' 2023-08-26T14:01:31.566 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ get_image_id cluster1 mirror image_46 2023-08-26T14:01:31.567 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local cluster=cluster1 2023-08-26T14:01:31.567 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local pool=mirror 2023-08-26T14:01:31.567 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local image=image_46 2023-08-26T14:01:31.567 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ rbd --cluster=cluster1 info mirror/image_46 2023-08-26T14:01:31.567 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2023-08-26T14:01:31.595 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:rbd: error opening image image_46: (2) No such file or directory 2023-08-26T14:01:31.597 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ image_id= 2023-08-26T14:01:31.598 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2023-08-26T14:01:31.598 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ sleep 0.1 2023-08-26T14:01:31.699 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ test_image_present cluster1 mirror image_46 deleted '' 2023-08-26T14:01:31.700 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster1 2023-08-26T14:01:31.700 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T14:01:31.700 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=image_46 2023-08-26T14:01:31.700 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local test_state=deleted 2023-08-26T14:01:31.700 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image_id= 2023-08-26T14:01:31.701 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local current_state=deleted 2023-08-26T14:01:31.701 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local current_image_id 2023-08-26T14:01:31.701 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ get_image_id cluster1 mirror image_46 2023-08-26T14:01:31.701 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local cluster=cluster1 2023-08-26T14:01:31.701 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local pool=mirror 2023-08-26T14:01:31.702 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local image=image_46 2023-08-26T14:01:31.702 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ rbd --cluster=cluster1 info mirror/image_46 2023-08-26T14:01:31.702 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2023-08-26T14:01:31.731 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:rbd: error opening image image_46: (2) No such file or directory 2023-08-26T14:01:31.735 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ current_image_id= 2023-08-26T14:01:31.735 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ test -n '' 2023-08-26T14:01:31.736 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ test deleted = deleted 2023-08-26T14:01:31.736 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ return 0 2023-08-26T14:01:31.736 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-08-26T14:01:31.736 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ image=image_47 2023-08-26T14:01:31.736 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ wait_for_image_present cluster1 mirror image_47 deleted 2023-08-26T14:01:31.736 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster1 2023-08-26T14:01:31.737 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T14:01:31.737 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=image_47 2023-08-26T14:01:31.737 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local state=deleted 2023-08-26T14:01:31.737 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image_id= 2023-08-26T14:01:31.737 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local s 2023-08-26T14:01:31.738 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ test -n '' 2023-08-26T14:01:31.738 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ get_image_id cluster1 mirror image_47 2023-08-26T14:01:31.738 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local cluster=cluster1 2023-08-26T14:01:31.738 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local pool=mirror 2023-08-26T14:01:31.738 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local image=image_47 2023-08-26T14:01:31.739 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ rbd --cluster=cluster1 info mirror/image_47 2023-08-26T14:01:31.739 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2023-08-26T14:01:31.766 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:rbd: error opening image image_47: (2) No such file or directory 2023-08-26T14:01:31.769 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ image_id= 2023-08-26T14:01:31.769 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2023-08-26T14:01:31.769 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ sleep 0.1 2023-08-26T14:01:31.870 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ test_image_present cluster1 mirror image_47 deleted '' 2023-08-26T14:01:31.871 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster1 2023-08-26T14:01:31.871 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T14:01:31.871 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=image_47 2023-08-26T14:01:31.871 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local test_state=deleted 2023-08-26T14:01:31.871 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image_id= 2023-08-26T14:01:31.872 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local current_state=deleted 2023-08-26T14:01:31.872 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local current_image_id 2023-08-26T14:01:31.872 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ get_image_id cluster1 mirror image_47 2023-08-26T14:01:31.872 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local cluster=cluster1 2023-08-26T14:01:31.872 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local pool=mirror 2023-08-26T14:01:31.873 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local image=image_47 2023-08-26T14:01:31.873 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ rbd --cluster=cluster1 info mirror/image_47 2023-08-26T14:01:31.873 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2023-08-26T14:01:31.904 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:rbd: error opening image image_47: (2) No such file or directory 2023-08-26T14:01:31.907 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ current_image_id= 2023-08-26T14:01:31.908 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ test -n '' 2023-08-26T14:01:31.908 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ test deleted = deleted 2023-08-26T14:01:31.908 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ return 0 2023-08-26T14:01:31.908 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-08-26T14:01:31.909 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ image=image_48 2023-08-26T14:01:31.909 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ wait_for_image_present cluster1 mirror image_48 deleted 2023-08-26T14:01:31.909 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster1 2023-08-26T14:01:31.909 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T14:01:31.909 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=image_48 2023-08-26T14:01:31.910 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local state=deleted 2023-08-26T14:01:31.910 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image_id= 2023-08-26T14:01:31.910 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local s 2023-08-26T14:01:31.910 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ test -n '' 2023-08-26T14:01:31.910 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ get_image_id cluster1 mirror image_48 2023-08-26T14:01:31.910 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local cluster=cluster1 2023-08-26T14:01:31.911 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local pool=mirror 2023-08-26T14:01:31.911 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local image=image_48 2023-08-26T14:01:31.911 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ rbd --cluster=cluster1 info mirror/image_48 2023-08-26T14:01:31.911 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2023-08-26T14:01:31.940 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:rbd: error opening image image_48: (2) No such file or directory 2023-08-26T14:01:31.942 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ image_id= 2023-08-26T14:01:31.943 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2023-08-26T14:01:31.943 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ sleep 0.1 2023-08-26T14:01:32.044 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ test_image_present cluster1 mirror image_48 deleted '' 2023-08-26T14:01:32.045 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster1 2023-08-26T14:01:32.045 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T14:01:32.045 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=image_48 2023-08-26T14:01:32.045 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local test_state=deleted 2023-08-26T14:01:32.045 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image_id= 2023-08-26T14:01:32.046 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local current_state=deleted 2023-08-26T14:01:32.046 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local current_image_id 2023-08-26T14:01:32.046 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ get_image_id cluster1 mirror image_48 2023-08-26T14:01:32.046 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local cluster=cluster1 2023-08-26T14:01:32.046 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local pool=mirror 2023-08-26T14:01:32.047 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local image=image_48 2023-08-26T14:01:32.047 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ rbd --cluster=cluster1 info mirror/image_48 2023-08-26T14:01:32.047 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2023-08-26T14:01:32.076 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:rbd: error opening image image_48: (2) No such file or directory 2023-08-26T14:01:32.080 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ current_image_id= 2023-08-26T14:01:32.081 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ test -n '' 2023-08-26T14:01:32.081 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ test deleted = deleted 2023-08-26T14:01:32.081 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ return 0 2023-08-26T14:01:32.081 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-08-26T14:01:32.081 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ image=image_49 2023-08-26T14:01:32.082 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ wait_for_image_present cluster1 mirror image_49 deleted 2023-08-26T14:01:32.082 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster1 2023-08-26T14:01:32.082 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T14:01:32.082 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=image_49 2023-08-26T14:01:32.082 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local state=deleted 2023-08-26T14:01:32.083 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image_id= 2023-08-26T14:01:32.083 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local s 2023-08-26T14:01:32.083 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ test -n '' 2023-08-26T14:01:32.083 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ get_image_id cluster1 mirror image_49 2023-08-26T14:01:32.083 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local cluster=cluster1 2023-08-26T14:01:32.084 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local pool=mirror 2023-08-26T14:01:32.084 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local image=image_49 2023-08-26T14:01:32.084 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ rbd --cluster=cluster1 info mirror/image_49 2023-08-26T14:01:32.084 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2023-08-26T14:01:32.111 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:rbd: error opening image image_49: (2) No such file or directory 2023-08-26T14:01:32.114 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ image_id= 2023-08-26T14:01:32.114 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2023-08-26T14:01:32.114 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ sleep 0.1 2023-08-26T14:01:32.216 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ test_image_present cluster1 mirror image_49 deleted '' 2023-08-26T14:01:32.216 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster1 2023-08-26T14:01:32.217 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T14:01:32.217 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=image_49 2023-08-26T14:01:32.217 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local test_state=deleted 2023-08-26T14:01:32.217 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image_id= 2023-08-26T14:01:32.217 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local current_state=deleted 2023-08-26T14:01:32.218 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local current_image_id 2023-08-26T14:01:32.218 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ get_image_id cluster1 mirror image_49 2023-08-26T14:01:32.218 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local cluster=cluster1 2023-08-26T14:01:32.218 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local pool=mirror 2023-08-26T14:01:32.218 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local image=image_49 2023-08-26T14:01:32.219 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ rbd --cluster=cluster1 info mirror/image_49 2023-08-26T14:01:32.219 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2023-08-26T14:01:32.247 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:rbd: error opening image image_49: (2) No such file or directory 2023-08-26T14:01:32.251 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ current_image_id= 2023-08-26T14:01:32.251 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ test -n '' 2023-08-26T14:01:32.251 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ test deleted = deleted 2023-08-26T14:01:32.252 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ return 0 2023-08-26T14:01:32.252 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2023-08-26T14:01:32.252 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ image=image_50 2023-08-26T14:01:32.252 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ wait_for_image_present cluster1 mirror image_50 deleted 2023-08-26T14:01:32.252 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster1 2023-08-26T14:01:32.253 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T14:01:32.253 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=image_50 2023-08-26T14:01:32.253 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local state=deleted 2023-08-26T14:01:32.253 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image_id= 2023-08-26T14:01:32.253 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local s 2023-08-26T14:01:32.254 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ test -n '' 2023-08-26T14:01:32.254 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ get_image_id cluster1 mirror image_50 2023-08-26T14:01:32.254 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local cluster=cluster1 2023-08-26T14:01:32.254 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local pool=mirror 2023-08-26T14:01:32.254 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local image=image_50 2023-08-26T14:01:32.255 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ rbd --cluster=cluster1 info mirror/image_50 2023-08-26T14:01:32.255 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2023-08-26T14:01:32.281 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:rbd: error opening image image_50: (2) No such file or directory 2023-08-26T14:01:32.284 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ image_id= 2023-08-26T14:01:32.284 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2023-08-26T14:01:32.284 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ sleep 0.1 2023-08-26T14:01:32.386 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ test_image_present cluster1 mirror image_50 deleted '' 2023-08-26T14:01:32.386 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster1 2023-08-26T14:01:32.387 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T14:01:32.387 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=image_50 2023-08-26T14:01:32.387 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local test_state=deleted 2023-08-26T14:01:32.387 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image_id= 2023-08-26T14:01:32.387 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local current_state=deleted 2023-08-26T14:01:32.388 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local current_image_id 2023-08-26T14:01:32.388 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ get_image_id cluster1 mirror image_50 2023-08-26T14:01:32.388 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local cluster=cluster1 2023-08-26T14:01:32.388 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local pool=mirror 2023-08-26T14:01:32.388 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local image=image_50 2023-08-26T14:01:32.388 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ rbd --cluster=cluster1 info mirror/image_50 2023-08-26T14:01:32.389 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2023-08-26T14:01:32.418 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:rbd: error opening image image_50: (2) No such file or directory 2023-08-26T14:01:32.421 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ current_image_id= 2023-08-26T14:01:32.421 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ test -n '' 2023-08-26T14:01:32.421 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ test deleted = deleted 2023-08-26T14:01:32.421 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ return 0 2023-08-26T14:01:32.422 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ testlog 'TEST: delete images during bootstrap' 2023-08-26T14:01:32.422 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ tee -a /tmp/tmp.TfDn6BcQ3N/rbd-mirror.test.log 2023-08-26T14:01:32.422 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ date '+%F %T' 2023-08-26T14:01:32.423 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ echo 2023-08-26 14:01:32 TEST: delete images during bootstrap 2023-08-26T14:01:32.423 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:2023-08-26 14:01:32 TEST: delete images during bootstrap 2023-08-26T14:01:32.423 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ set_pool_mirror_mode cluster1 mirror image 2023-08-26T14:01:32.424 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster1 2023-08-26T14:01:32.424 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T14:01:32.424 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local mode=image 2023-08-26T14:01:32.424 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rbd --cluster=cluster1 mirror pool enable mirror image 2023-08-26T14:01:32.454 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:rbd: mirroring is already configured for image mode 2023-08-26T14:01:32.457 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ set_pool_mirror_mode cluster2 mirror image 2023-08-26T14:01:32.457 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster2 2023-08-26T14:01:32.457 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T14:01:32.457 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local mode=image 2023-08-26T14:01:32.458 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rbd --cluster=cluster2 mirror pool enable mirror image 2023-08-26T14:01:32.487 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:rbd: mirroring is already configured for image mode 2023-08-26T14:01:32.491 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ start_mirror cluster1 2023-08-26T14:01:32.492 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster1 2023-08-26T14:01:32.492 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local instance 2023-08-26T14:01:32.492 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ set_cluster_instance cluster1 cluster instance 2023-08-26T14:01:32.492 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local val=cluster1 2023-08-26T14:01:32.492 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster_var_name=cluster 2023-08-26T14:01:32.493 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local instance_var_name=instance 2023-08-26T14:01:32.493 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ cluster=cluster1 2023-08-26T14:01:32.493 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ instance=cluster1 2023-08-26T14:01:32.493 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ '[' cluster1 = cluster1 ']' 2023-08-26T14:01:32.493 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ instance=0 2023-08-26T14:01:32.494 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ eval cluster=cluster1 2023-08-26T14:01:32.494 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ cluster=cluster1 2023-08-26T14:01:32.494 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ eval instance=0 2023-08-26T14:01:32.494 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ instance=0 2023-08-26T14:01:32.494 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ test -n 1 2023-08-26T14:01:32.495 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ return 2023-08-26T14:01:32.495 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ image=test 2023-08-26T14:01:32.495 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ seq 1 10 2023-08-26T14:01:32.495 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ for i in `seq 1 10` 2023-08-26T14:01:32.495 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ image=image_1 2023-08-26T14:01:32.495 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ create_image cluster2 mirror image_1 512M 2023-08-26T14:01:32.496 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster2 2023-08-26T14:01:32.496 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ shift 2023-08-26T14:01:32.496 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T14:01:32.496 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ shift 2023-08-26T14:01:32.496 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=image_1 2023-08-26T14:01:32.497 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ shift 2023-08-26T14:01:32.497 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local size=128 2023-08-26T14:01:32.497 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ '[' -n 512M ']' 2023-08-26T14:01:32.497 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ size=512M 2023-08-26T14:01:32.497 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ shift 2023-08-26T14:01:32.497 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rbd --cluster cluster2 create --size 512M --image-feature layering,exclusive-lock mirror/image_1 2023-08-26T14:01:32.538 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ enable_mirror cluster2 mirror image_1 2023-08-26T14:01:32.538 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster2 2023-08-26T14:01:32.538 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T14:01:32.538 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=image_1 2023-08-26T14:01:32.538 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local mode=snapshot 2023-08-26T14:01:32.539 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_1 snapshot 2023-08-26T14:01:33.360 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:Mirroring enabled 2023-08-26T14:01:33.368 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rbd --cluster=cluster2 info mirror/image_1 2023-08-26T14:01:33.405 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:rbd image 'image_1': 2023-08-26T14:01:33.405 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: size 512 MiB in 128 objects 2023-08-26T14:01:33.405 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: order 22 (4 MiB objects) 2023-08-26T14:01:33.406 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: snapshot_count: 1 2023-08-26T14:01:33.406 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: id: 186c5c34b48a 2023-08-26T14:01:33.406 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: block_name_prefix: rbd_data.186c5c34b48a 2023-08-26T14:01:33.406 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: format: 2 2023-08-26T14:01:33.406 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: features: layering, exclusive-lock 2023-08-26T14:01:33.407 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: op_features: 2023-08-26T14:01:33.407 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: flags: 2023-08-26T14:01:33.407 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: create_timestamp: Sat Aug 26 14:01:32 2023 2023-08-26T14:01:33.407 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: access_timestamp: Sat Aug 26 14:01:32 2023 2023-08-26T14:01:33.407 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: modify_timestamp: Sat Aug 26 14:01:32 2023 2023-08-26T14:01:33.408 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: mirroring state: enabled 2023-08-26T14:01:33.408 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: mirroring mode: snapshot 2023-08-26T14:01:33.408 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: mirroring global id: 62ee6ec9-d2e5-48ec-a9ce-d6b188191d43 2023-08-26T14:01:33.408 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: mirroring primary: true 2023-08-26T14:01:33.409 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ stress_write_image cluster2 mirror image_1 2023-08-26T14:01:33.409 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster2 2023-08-26T14:01:33.410 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T14:01:33.410 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=image_1 2023-08-26T14:01:33.410 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2023-08-26T14:01:33.410 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local duration=13 2023-08-26T14:01:33.410 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ set +e 2023-08-26T14:01:33.411 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ timeout 13s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_1 --debug-rbd=20 --debug-journaler=20 2023-08-26T14:01:33.645 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:100 IOs, 49199104 bytes 2023-08-26T14:01:33.838 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:200 IOs, 113053696 bytes 2023-08-26T14:01:34.138 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:300 IOs, 198581248 bytes 2023-08-26T14:01:34.481 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:400 IOs, 306733056 bytes 2023-08-26T14:01:34.627 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:500 IOs, 326336512 bytes 2023-08-26T14:01:34.901 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:600 IOs, 411079680 bytes 2023-08-26T14:01:35.070 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:700 IOs, 457938944 bytes 2023-08-26T14:01:35.348 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:800 IOs, 544968704 bytes 2023-08-26T14:01:35.559 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:900 IOs, 589665280 bytes 2023-08-26T14:01:35.761 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:1000 IOs, 630132736 bytes 2023-08-26T14:01:36.025 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:1100 IOs, 712642560 bytes 2023-08-26T14:01:36.130 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:1200 IOs, 717563904 bytes 2023-08-26T14:01:36.307 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:1300 IOs, 753612800 bytes 2023-08-26T14:01:36.423 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:1400 IOs, 768207872 bytes 2023-08-26T14:01:36.628 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:1500 IOs, 812933120 bytes 2023-08-26T14:01:36.961 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:1600 IOs, 887963648 bytes 2023-08-26T14:01:37.136 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:1700 IOs, 919962624 bytes 2023-08-26T14:01:37.309 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:1800 IOs, 936682496 bytes 2023-08-26T14:01:37.436 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:1900 IOs, 949118976 bytes 2023-08-26T14:01:37.602 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:2000 IOs, 981214208 bytes 2023-08-26T14:01:37.911 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:2100 IOs, 1082888192 bytes 2023-08-26T14:01:38.114 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:2200 IOs, 1133417472 bytes 2023-08-26T14:01:38.278 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:2300 IOs, 1167284224 bytes 2023-08-26T14:01:38.458 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:2400 IOs, 1222183936 bytes 2023-08-26T14:01:38.691 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:2500 IOs, 1272099840 bytes 2023-08-26T14:01:38.859 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:2600 IOs, 1318238208 bytes 2023-08-26T14:01:39.049 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:2700 IOs, 1348312064 bytes 2023-08-26T14:01:39.221 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:2800 IOs, 1382382592 bytes 2023-08-26T14:01:39.412 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:2900 IOs, 1423144960 bytes 2023-08-26T14:01:39.567 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:3000 IOs, 1470296064 bytes 2023-08-26T14:01:39.732 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:3100 IOs, 1479578624 bytes 2023-08-26T14:01:39.939 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:3200 IOs, 1532097536 bytes 2023-08-26T14:01:40.106 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:3300 IOs, 1558453248 bytes 2023-08-26T14:01:40.265 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:3400 IOs, 1583126528 bytes 2023-08-26T14:01:40.406 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:3500 IOs, 1612398592 bytes 2023-08-26T14:01:40.547 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:3600 IOs, 1613001728 bytes 2023-08-26T14:01:40.760 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:3700 IOs, 1672291328 bytes 2023-08-26T14:01:40.994 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:3800 IOs, 1738874880 bytes 2023-08-26T14:01:41.207 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:3900 IOs, 1796928512 bytes 2023-08-26T14:01:41.403 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:4000 IOs, 1844758528 bytes 2023-08-26T14:01:41.616 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:4100 IOs, 1913868288 bytes 2023-08-26T14:01:41.867 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:4200 IOs, 1960056832 bytes 2023-08-26T14:01:42.145 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:4300 IOs, 2049385472 bytes 2023-08-26T14:01:42.304 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:4400 IOs, 2067843072 bytes 2023-08-26T14:01:42.476 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:4500 IOs, 2095995904 bytes 2023-08-26T14:01:42.713 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:4600 IOs, 2157931520 bytes 2023-08-26T14:01:42.909 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:4700 IOs, 2199608320 bytes 2023-08-26T14:01:43.260 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:4800 IOs, 2325537792 bytes 2023-08-26T14:01:43.449 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:4900 IOs, 2380892160 bytes 2023-08-26T14:01:43.587 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:5000 IOs, 2381508608 bytes 2023-08-26T14:01:43.743 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:5100 IOs, 2404658176 bytes 2023-08-26T14:01:43.951 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:5200 IOs, 2454085632 bytes 2023-08-26T14:01:44.138 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:5300 IOs, 2497482752 bytes 2023-08-26T14:01:44.332 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:5400 IOs, 2545839104 bytes 2023-08-26T14:01:44.482 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:5500 IOs, 2572410880 bytes 2023-08-26T14:01:44.773 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:5600 IOs, 2667537408 bytes 2023-08-26T14:01:44.887 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:5700 IOs, 2674545664 bytes 2023-08-26T14:01:45.030 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:5800 IOs, 2691491840 bytes 2023-08-26T14:01:45.209 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:5900 IOs, 2725811200 bytes 2023-08-26T14:01:45.347 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:6000 IOs, 2743206912 bytes 2023-08-26T14:01:45.502 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:6100 IOs, 2771402752 bytes 2023-08-26T14:01:45.737 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:6200 IOs, 2853393408 bytes 2023-08-26T14:01:45.994 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:6300 IOs, 2910332928 bytes 2023-08-26T14:01:46.163 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:6400 IOs, 2945251328 bytes 2023-08-26T14:01:46.317 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:6500 IOs, 2969734144 bytes 2023-08-26T14:01:46.419 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ error_code=124 2023-08-26T14:01:46.419 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ set -e 2023-08-26T14:01:46.420 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ '[' 124 -eq 124 ']' 2023-08-26T14:01:46.420 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ return 0 2023-08-26T14:01:46.420 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ wait_for_image_present cluster1 mirror image_1 present 2023-08-26T14:01:46.420 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster1 2023-08-26T14:01:46.420 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T14:01:46.421 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=image_1 2023-08-26T14:01:46.421 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local state=present 2023-08-26T14:01:46.421 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image_id= 2023-08-26T14:01:46.421 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local s 2023-08-26T14:01:46.421 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ test -n '' 2023-08-26T14:01:46.422 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ get_image_id cluster1 mirror image_1 2023-08-26T14:01:46.422 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local cluster=cluster1 2023-08-26T14:01:46.422 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local pool=mirror 2023-08-26T14:01:46.422 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local image=image_1 2023-08-26T14:01:46.422 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ rbd --cluster=cluster1 info mirror/image_1 2023-08-26T14:01:46.423 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2023-08-26T14:01:46.465 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ image_id=12c068a91df3 2023-08-26T14:01:46.465 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2023-08-26T14:01:46.465 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ sleep 0.1 2023-08-26T14:01:46.566 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ test_image_present cluster1 mirror image_1 present 12c068a91df3 2023-08-26T14:01:46.567 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster1 2023-08-26T14:01:46.567 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T14:01:46.567 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=image_1 2023-08-26T14:01:46.567 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local test_state=present 2023-08-26T14:01:46.568 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image_id=12c068a91df3 2023-08-26T14:01:46.568 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local current_state=deleted 2023-08-26T14:01:46.568 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local current_image_id 2023-08-26T14:01:46.568 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ get_image_id cluster1 mirror image_1 2023-08-26T14:01:46.568 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local cluster=cluster1 2023-08-26T14:01:46.569 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local pool=mirror 2023-08-26T14:01:46.569 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local image=image_1 2023-08-26T14:01:46.569 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ rbd --cluster=cluster1 info mirror/image_1 2023-08-26T14:01:46.569 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2023-08-26T14:01:46.609 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ current_image_id=12c068a91df3 2023-08-26T14:01:46.610 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ test -n 12c068a91df3 2023-08-26T14:01:46.610 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ test -z 12c068a91df3 -o 12c068a91df3 = 12c068a91df3 2023-08-26T14:01:46.610 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ current_state=present 2023-08-26T14:01:46.610 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ test present = present 2023-08-26T14:01:46.610 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ return 0 2023-08-26T14:01:46.611 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ disable_mirror cluster2 mirror image_1 2023-08-26T14:01:46.611 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster2 2023-08-26T14:01:46.611 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T14:01:46.611 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=image_1 2023-08-26T14:01:46.611 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rbd --cluster=cluster2 mirror image disable mirror/image_1 2023-08-26T14:01:47.340 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi124.stderr:2023-08-26T14:01:47.338+0000 7f0338b37700 -1 rbd::mirror::image_replayer::snapshot::Replayer: 0x5579900cc400 scan_remote_mirror_snapshots: failed to locate remote start snapshot: snap_id=368 2023-08-26T14:01:47.341 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi124.stderr:2023-08-26T14:01:47.338+0000 7f0338b37700 -1 rbd::mirror::image_replayer::snapshot::Replayer: 0x5579900cc400 scan_remote_mirror_snapshots: split-brain detected: failed to find matching non-primary snapshot in remote image: local_snap_id_start=368, local_snap_ns=[mirror state=non-primary, complete=1, mirror_peer_uuids=, primary_mirror_uuid=d88cafc5-6d65-4864-9230-fe3ef515f83d, primary_snap_id=170, last_copied_object_number=128, snap_seqs={368=18446744073709551614}] 2023-08-26T14:01:47.617 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi124.stderr:2023-08-26T14:01:47.615+0000 7f6c32e39700 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x5621b9ee0820 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2023-08-26T14:01:47.617 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi124.stderr:2023-08-26T14:01:47.615+0000 7f6c39646700 -1 rbd::mirror::ImageReplayer: 0x5621ba457400 [3/62ee6ec9-d2e5-48ec-a9ce-d6b188191d43] operator(): start failed: (85) Interrupted system call should be restarted 2023-08-26T14:01:51.551 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.0 with skip (0.3300485690035232) > weight (0.25) 2023-08-26T14:01:51.551 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.1 with skip (0.4396717151484123) > weight (0.25) 2023-08-26T14:01:51.551 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.2 with skip (0.6298663429079421) > weight (0.25) 2023-08-26T14:01:51.552 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.3 with skip (0.8661085443732104) > weight (0.25) 2023-08-26T14:01:51.552 INFO:tasks.rbd_mirror_thrash:waiting for 99 secs before thrashing 2023-08-26T14:01:52.340 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:Mirroring disabled 2023-08-26T14:01:52.349 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ wait_for_image_present cluster1 mirror image_1 deleted 2023-08-26T14:01:52.349 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster1 2023-08-26T14:01:52.349 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T14:01:52.349 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=image_1 2023-08-26T14:01:52.349 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local state=deleted 2023-08-26T14:01:52.350 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image_id= 2023-08-26T14:01:52.350 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local s 2023-08-26T14:01:52.350 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ test -n '' 2023-08-26T14:01:52.350 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ get_image_id cluster1 mirror image_1 2023-08-26T14:01:52.350 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local cluster=cluster1 2023-08-26T14:01:52.351 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local pool=mirror 2023-08-26T14:01:52.351 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local image=image_1 2023-08-26T14:01:52.351 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ rbd --cluster=cluster1 info mirror/image_1 2023-08-26T14:01:52.351 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2023-08-26T14:01:52.387 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:rbd: error opening image image_1: (2) No such file or directory 2023-08-26T14:01:52.391 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ image_id= 2023-08-26T14:01:52.391 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2023-08-26T14:01:52.391 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ sleep 0.1 2023-08-26T14:01:52.493 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ test_image_present cluster1 mirror image_1 deleted '' 2023-08-26T14:01:52.493 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster1 2023-08-26T14:01:52.493 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T14:01:52.494 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=image_1 2023-08-26T14:01:52.494 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local test_state=deleted 2023-08-26T14:01:52.494 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image_id= 2023-08-26T14:01:52.494 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local current_state=deleted 2023-08-26T14:01:52.494 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local current_image_id 2023-08-26T14:01:52.495 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ get_image_id cluster1 mirror image_1 2023-08-26T14:01:52.495 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local cluster=cluster1 2023-08-26T14:01:52.495 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local pool=mirror 2023-08-26T14:01:52.495 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local image=image_1 2023-08-26T14:01:52.495 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ rbd --cluster=cluster1 info mirror/image_1 2023-08-26T14:01:52.496 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2023-08-26T14:01:52.531 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:rbd: error opening image image_1: (2) No such file or directory 2023-08-26T14:01:52.535 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ current_image_id= 2023-08-26T14:01:52.535 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ test -n '' 2023-08-26T14:01:52.536 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ test deleted = deleted 2023-08-26T14:01:52.536 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ return 0 2023-08-26T14:01:52.536 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ purge_snapshots cluster2 mirror image_1 2023-08-26T14:01:52.536 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster2 2023-08-26T14:01:52.536 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T14:01:52.537 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=image_1 2023-08-26T14:01:52.537 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rbd --cluster cluster2 snap purge mirror/image_1 2023-08-26T14:01:52.588 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ remove_image_retry cluster2 mirror image_1 2023-08-26T14:01:52.588 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster2 2023-08-26T14:01:52.589 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T14:01:52.589 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=image_1 2023-08-26T14:01:52.589 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ for s in 0 1 2 4 8 16 32 2023-08-26T14:01:52.589 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ sleep 0 2023-08-26T14:01:52.590 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ remove_image cluster2 mirror image_1 2023-08-26T14:01:52.590 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster2 2023-08-26T14:01:52.590 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T14:01:52.590 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=image_1 2023-08-26T14:01:52.591 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_1 2023-08-26T14:01:52.636 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rbd --cluster=cluster2 rm mirror/image_1 2023-08-26T14:02:15.493 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr: Removing image: 1% complete... Removing image: 2% complete... Removing image: 3% complete... Removing image: 4% complete... Removing image: 5% complete... Removing image: 6% complete... Removing image: 7% complete... Removing image: 8% complete... Removing image: 9% complete... Removing image: 10% complete... Removing image: 11% complete... Removing image: 12% complete... Removing image: 13% complete... Removing image: 14% complete... Removing image: 15% complete... Removing image: 16% complete... Removing image: 17% complete... Removing image: 18% complete... Removing image: 19% complete... Removing image: 20% complete... Removing image: 21% complete... Removing image: 22% complete... Removing image: 23% complete... Removing image: 24% complete... Removing image: 25% complete... Removing image: 26% complete... Removing image: 27% complete... Removing image: 28% complete... Removing image: 29% complete... Removing image: 30% complete... Removing image: 31% complete... Removing image: 32% complete... Removing image: 33% complete... Removing image: 34% complete... Removing image: 35% complete... Removing image: 36% complete... Removing image: 37% complete... Removing image: 38% complete... Removing image: 39% complete... Removing image: 40% complete... Removing image: 41% complete... Removing image: 42% complete... Removing image: 43% complete... Removing image: 44% complete... Removing image: 45% complete... Removing image: 46% complete... Removing image: 47% complete... Removing image: 48% complete... Removing image: 49% complete... Removing image: 50% complete... Removing image: 51% complete... Removing image: 52% complete... Removing image: 53% complete... Removing image: 54% complete... Removing image: 55% complete... Removing image: 56% complete... Removing image: 57% complete... Removing image: 58% complete... Removing image: 59% complete... Removing image: 60% complete... Removing image: 61% complete... Removing image: 62% complete... Removing image: 63% complete... Removing image: 64% complete... Removing image: 65% complete... Removing image: 66% complete... Removing image: 67% complete... Removing image: 68% complete... Removing image: 69% complete... Removing image: 70% complete... Removing image: 71% complete... Removing image: 72% complete... Removing image: 73% complete... Removing image: 74% complete... Removing image: 75% complete... Removing image: 76% complete... Removing image: 77% complete... Removing image: 78% complete... Removing image: 79% complete... Removing image: 80% complete... Removing image: 81% complete... Removing image: 82% complete... Removing image: 83% complete... Removing image: 84% complete... Removing image: 85% complete... Removing image: 86% complete... Removing image: 87% complete... Removing image: 88% complete... Removing image: 89% complete... Removing image: 90% complete... Removing image: 91% complete... Removing image: 92% complete... Removing image: 93% complete... Removing image: 94% complete... Removing image: 95% complete... Removing image: 96% complete... Removing image: 97% complete... Removing image: 98% complete... Removing image: 99% complete... Removing image: 100% complete...done. 2023-08-26T14:02:15.497 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ return 0 2023-08-26T14:02:15.497 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ for i in `seq 1 10` 2023-08-26T14:02:15.497 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ image=image_2 2023-08-26T14:02:15.497 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ create_image cluster2 mirror image_2 512M 2023-08-26T14:02:15.497 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster2 2023-08-26T14:02:15.498 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ shift 2023-08-26T14:02:15.498 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T14:02:15.498 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ shift 2023-08-26T14:02:15.498 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=image_2 2023-08-26T14:02:15.498 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ shift 2023-08-26T14:02:15.499 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local size=128 2023-08-26T14:02:15.499 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ '[' -n 512M ']' 2023-08-26T14:02:15.499 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ size=512M 2023-08-26T14:02:15.499 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ shift 2023-08-26T14:02:15.499 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rbd --cluster cluster2 create --size 512M --image-feature layering,exclusive-lock mirror/image_2 2023-08-26T14:02:15.543 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ enable_mirror cluster2 mirror image_2 2023-08-26T14:02:15.543 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster2 2023-08-26T14:02:15.543 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T14:02:15.543 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=image_2 2023-08-26T14:02:15.544 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local mode=snapshot 2023-08-26T14:02:15.544 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_2 snapshot 2023-08-26T14:02:16.274 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:Mirroring enabled 2023-08-26T14:02:16.282 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rbd --cluster=cluster2 info mirror/image_2 2023-08-26T14:02:16.319 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:rbd image 'image_2': 2023-08-26T14:02:16.320 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: size 512 MiB in 128 objects 2023-08-26T14:02:16.320 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: order 22 (4 MiB objects) 2023-08-26T14:02:16.320 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: snapshot_count: 1 2023-08-26T14:02:16.320 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: id: 187da0f4d37c 2023-08-26T14:02:16.320 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: block_name_prefix: rbd_data.187da0f4d37c 2023-08-26T14:02:16.321 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: format: 2 2023-08-26T14:02:16.321 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: features: layering, exclusive-lock 2023-08-26T14:02:16.321 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: op_features: 2023-08-26T14:02:16.321 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: flags: 2023-08-26T14:02:16.321 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: create_timestamp: Sat Aug 26 14:02:15 2023 2023-08-26T14:02:16.322 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: access_timestamp: Sat Aug 26 14:02:15 2023 2023-08-26T14:02:16.322 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: modify_timestamp: Sat Aug 26 14:02:15 2023 2023-08-26T14:02:16.322 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: mirroring state: enabled 2023-08-26T14:02:16.322 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: mirroring mode: snapshot 2023-08-26T14:02:16.322 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: mirroring global id: 518061f6-f8d3-4a49-b886-5f58c1f48937 2023-08-26T14:02:16.323 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: mirroring primary: true 2023-08-26T14:02:16.323 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ stress_write_image cluster2 mirror image_2 2023-08-26T14:02:16.323 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster2 2023-08-26T14:02:16.324 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T14:02:16.324 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=image_2 2023-08-26T14:02:16.324 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2023-08-26T14:02:16.325 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local duration=13 2023-08-26T14:02:16.325 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ set +e 2023-08-26T14:02:16.325 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ timeout 13s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_2 --debug-rbd=20 --debug-journaler=20 2023-08-26T14:02:16.500 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:100 IOs, 17106944 bytes 2023-08-26T14:02:16.709 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:200 IOs, 75616256 bytes 2023-08-26T14:02:16.816 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:300 IOs, 76213248 bytes 2023-08-26T14:02:16.932 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:400 IOs, 84982784 bytes 2023-08-26T14:02:17.044 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:500 IOs, 85574656 bytes 2023-08-26T14:02:17.315 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:600 IOs, 166568960 bytes 2023-08-26T14:02:17.561 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:700 IOs, 245163008 bytes 2023-08-26T14:02:17.708 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:800 IOs, 267556864 bytes 2023-08-26T14:02:17.910 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:900 IOs, 316080128 bytes 2023-08-26T14:02:18.073 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:1000 IOs, 344181760 bytes 2023-08-26T14:02:18.278 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:1100 IOs, 392124416 bytes 2023-08-26T14:02:18.462 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:1200 IOs, 426181632 bytes 2023-08-26T14:02:18.574 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:1300 IOs, 434599936 bytes 2023-08-26T14:02:18.671 INFO:tasks.ceph.cluster2.osd.0.smithi124.stderr:2023-08-26T14:02:18.669+0000 7fb612674700 -1 bdev(0x56452769a000 /var/lib/ceph/osd/cluster2-0/block) _aio_thread got r=-28 ((28) No space left on device) 2023-08-26T14:02:18.673 INFO:tasks.ceph.cluster2.osd.0.smithi124.stderr:/home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/18.0.0-5706-ge7eb9853/rpm/el8/BUILD/ceph-18.0.0-5706-ge7eb9853/src/blk/kernel/KernelDevice.cc: In function 'void KernelDevice::_aio_thread()' thread 7fb612674700 time 2023-08-26T14:02:18.670361+0000 2023-08-26T14:02:18.673 INFO:tasks.ceph.cluster2.osd.0.smithi124.stderr:/home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/18.0.0-5706-ge7eb9853/rpm/el8/BUILD/ceph-18.0.0-5706-ge7eb9853/src/blk/kernel/KernelDevice.cc: 633: ceph_abort_msg("Unexpected IO error. This may suggest a hardware issue. Please check your kernel log!") 2023-08-26T14:02:18.674 INFO:tasks.ceph.cluster2.osd.0.smithi124.stderr: ceph version 18.0.0-5706-ge7eb9853 (e7eb9853fddaf661c68f207346131a7e00d8ed2f) reef (dev) 2023-08-26T14:02:18.674 INFO:tasks.ceph.cluster2.osd.0.smithi124.stderr: 1: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string, std::allocator > const&)+0xd7) [0x56452487d9b5] 2023-08-26T14:02:18.674 INFO:tasks.ceph.cluster2.osd.0.smithi124.stderr: 2: (KernelDevice::_aio_thread()+0x1686) [0x5645254a7416] 2023-08-26T14:02:18.675 INFO:tasks.ceph.cluster2.osd.0.smithi124.stderr: 3: (KernelDevice::AioCompletionThread::entry()+0x11) [0x5645254aced1] 2023-08-26T14:02:18.675 INFO:tasks.ceph.cluster2.osd.0.smithi124.stderr: 4: /lib64/libpthread.so.0(+0x81cf) [0x7fb62519b1cf] 2023-08-26T14:02:18.675 INFO:tasks.ceph.cluster2.osd.0.smithi124.stderr: 5: clone() 2023-08-26T14:02:18.675 INFO:tasks.ceph.cluster2.osd.0.smithi124.stderr:2023-08-26T14:02:18.671+0000 7fb612674700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/18.0.0-5706-ge7eb9853/rpm/el8/BUILD/ceph-18.0.0-5706-ge7eb9853/src/blk/kernel/KernelDevice.cc: In function 'void KernelDevice::_aio_thread()' thread 7fb612674700 time 2023-08-26T14:02:18.670361+0000 2023-08-26T14:02:18.675 INFO:tasks.ceph.cluster2.osd.0.smithi124.stderr:/home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/18.0.0-5706-ge7eb9853/rpm/el8/BUILD/ceph-18.0.0-5706-ge7eb9853/src/blk/kernel/KernelDevice.cc: 633: ceph_abort_msg("Unexpected IO error. This may suggest a hardware issue. Please check your kernel log!") 2023-08-26T14:02:18.676 INFO:tasks.ceph.cluster2.osd.0.smithi124.stderr: 2023-08-26T14:02:18.676 INFO:tasks.ceph.cluster2.osd.0.smithi124.stderr: ceph version 18.0.0-5706-ge7eb9853 (e7eb9853fddaf661c68f207346131a7e00d8ed2f) reef (dev) 2023-08-26T14:02:18.676 INFO:tasks.ceph.cluster2.osd.0.smithi124.stderr: 1: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string, std::allocator > const&)+0xd7) [0x56452487d9b5] 2023-08-26T14:02:18.676 INFO:tasks.ceph.cluster2.osd.0.smithi124.stderr: 2: (KernelDevice::_aio_thread()+0x1686) [0x5645254a7416] 2023-08-26T14:02:18.676 INFO:tasks.ceph.cluster2.osd.0.smithi124.stderr: 3: (KernelDevice::AioCompletionThread::entry()+0x11) [0x5645254aced1] 2023-08-26T14:02:18.677 INFO:tasks.ceph.cluster2.osd.0.smithi124.stderr: 4: /lib64/libpthread.so.0(+0x81cf) [0x7fb62519b1cf] 2023-08-26T14:02:18.677 INFO:tasks.ceph.cluster2.osd.0.smithi124.stderr: 5: clone() 2023-08-26T14:02:18.677 INFO:tasks.ceph.cluster2.osd.0.smithi124.stderr: 2023-08-26T14:02:18.677 INFO:tasks.ceph.cluster2.osd.0.smithi124.stderr:*** Caught signal (Aborted) ** 2023-08-26T14:02:18.678 INFO:tasks.ceph.cluster2.osd.0.smithi124.stderr: in thread 7fb612674700 thread_name:bstore_aio 2023-08-26T14:02:18.678 INFO:tasks.ceph.cluster2.osd.0.smithi124.stderr: ceph version 18.0.0-5706-ge7eb9853 (e7eb9853fddaf661c68f207346131a7e00d8ed2f) reef (dev) 2023-08-26T14:02:18.678 INFO:tasks.ceph.cluster2.osd.0.smithi124.stderr: 1: /lib64/libpthread.so.0(+0x12ce0) [0x7fb6251a5ce0] 2023-08-26T14:02:18.679 INFO:tasks.ceph.cluster2.osd.0.smithi124.stderr: 2: gsignal() 2023-08-26T14:02:18.679 INFO:tasks.ceph.cluster2.osd.0.smithi124.stderr: 3: abort() 2023-08-26T14:02:18.679 INFO:tasks.ceph.cluster2.osd.0.smithi124.stderr: 4: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string, std::allocator > const&)+0x197) [0x56452487da75] 2023-08-26T14:02:18.679 INFO:tasks.ceph.cluster2.osd.0.smithi124.stderr: 5: (KernelDevice::_aio_thread()+0x1686) [0x5645254a7416] 2023-08-26T14:02:18.680 INFO:tasks.ceph.cluster2.osd.0.smithi124.stderr: 6: (KernelDevice::AioCompletionThread::entry()+0x11) [0x5645254aced1] 2023-08-26T14:02:18.680 INFO:tasks.ceph.cluster2.osd.0.smithi124.stderr: 7: /lib64/libpthread.so.0(+0x81cf) [0x7fb62519b1cf] 2023-08-26T14:02:18.680 INFO:tasks.ceph.cluster2.osd.0.smithi124.stderr: 8: clone() 2023-08-26T14:02:18.680 INFO:tasks.ceph.cluster2.osd.0.smithi124.stderr:2023-08-26T14:02:18.673+0000 7fb612674700 -1 *** Caught signal (Aborted) ** 2023-08-26T14:02:18.680 INFO:tasks.ceph.cluster2.osd.0.smithi124.stderr: in thread 7fb612674700 thread_name:bstore_aio 2023-08-26T14:02:18.681 INFO:tasks.ceph.cluster2.osd.0.smithi124.stderr: 2023-08-26T14:02:18.681 INFO:tasks.ceph.cluster2.osd.0.smithi124.stderr: ceph version 18.0.0-5706-ge7eb9853 (e7eb9853fddaf661c68f207346131a7e00d8ed2f) reef (dev) 2023-08-26T14:02:18.681 INFO:tasks.ceph.cluster2.osd.0.smithi124.stderr: 1: /lib64/libpthread.so.0(+0x12ce0) [0x7fb6251a5ce0] 2023-08-26T14:02:18.681 INFO:tasks.ceph.cluster2.osd.0.smithi124.stderr: 2: gsignal() 2023-08-26T14:02:18.681 INFO:tasks.ceph.cluster2.osd.0.smithi124.stderr: 3: abort() 2023-08-26T14:02:18.682 INFO:tasks.ceph.cluster2.osd.0.smithi124.stderr: 4: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string, std::allocator > const&)+0x197) [0x56452487da75] 2023-08-26T14:02:18.682 INFO:tasks.ceph.cluster2.osd.0.smithi124.stderr: 5: (KernelDevice::_aio_thread()+0x1686) [0x5645254a7416] 2023-08-26T14:02:18.682 INFO:tasks.ceph.cluster2.osd.0.smithi124.stderr: 6: (KernelDevice::AioCompletionThread::entry()+0x11) [0x5645254aced1] 2023-08-26T14:02:18.682 INFO:tasks.ceph.cluster2.osd.0.smithi124.stderr: 7: /lib64/libpthread.so.0(+0x81cf) [0x7fb62519b1cf] 2023-08-26T14:02:18.682 INFO:tasks.ceph.cluster2.osd.0.smithi124.stderr: 8: clone() 2023-08-26T14:02:18.683 INFO:tasks.ceph.cluster2.osd.0.smithi124.stderr: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2023-08-26T14:02:18.683 INFO:tasks.ceph.cluster2.osd.0.smithi124.stderr: 2023-08-26T14:02:18.692 INFO:tasks.ceph.cluster2.osd.0.smithi124.stderr: -256> 2023-08-26T14:02:18.669+0000 7fb612674700 -1 bdev(0x56452769a000 /var/lib/ceph/osd/cluster2-0/block) _aio_thread got r=-28 ((28) No space left on device) 2023-08-26T14:02:18.693 INFO:tasks.ceph.cluster2.osd.0.smithi124.stderr: -129> 2023-08-26T14:02:18.671+0000 7fb612674700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/18.0.0-5706-ge7eb9853/rpm/el8/BUILD/ceph-18.0.0-5706-ge7eb9853/src/blk/kernel/KernelDevice.cc: In function 'void KernelDevice::_aio_thread()' thread 7fb612674700 time 2023-08-26T14:02:18.670361+0000 2023-08-26T14:02:18.693 INFO:tasks.ceph.cluster2.osd.0.smithi124.stderr:/home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/18.0.0-5706-ge7eb9853/rpm/el8/BUILD/ceph-18.0.0-5706-ge7eb9853/src/blk/kernel/KernelDevice.cc: 633: ceph_abort_msg("Unexpected IO error. This may suggest a hardware issue. Please check your kernel log!") 2023-08-26T14:02:18.693 INFO:tasks.ceph.cluster2.osd.0.smithi124.stderr: 2023-08-26T14:02:18.693 INFO:tasks.ceph.cluster2.osd.0.smithi124.stderr: ceph version 18.0.0-5706-ge7eb9853 (e7eb9853fddaf661c68f207346131a7e00d8ed2f) reef (dev) 2023-08-26T14:02:18.693 INFO:tasks.ceph.cluster2.osd.0.smithi124.stderr: 1: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string, std::allocator > const&)+0xd7) [0x56452487d9b5] 2023-08-26T14:02:18.694 INFO:tasks.ceph.cluster2.osd.0.smithi124.stderr: 2: (KernelDevice::_aio_thread()+0x1686) [0x5645254a7416] 2023-08-26T14:02:18.694 INFO:tasks.ceph.cluster2.osd.0.smithi124.stderr: 3: (KernelDevice::AioCompletionThread::entry()+0x11) [0x5645254aced1] 2023-08-26T14:02:18.694 INFO:tasks.ceph.cluster2.osd.0.smithi124.stderr: 4: /lib64/libpthread.so.0(+0x81cf) [0x7fb62519b1cf] 2023-08-26T14:02:18.694 INFO:tasks.ceph.cluster2.osd.0.smithi124.stderr: 5: clone() 2023-08-26T14:02:18.694 INFO:tasks.ceph.cluster2.osd.0.smithi124.stderr: 2023-08-26T14:02:18.695 INFO:tasks.ceph.cluster2.osd.0.smithi124.stderr: 0> 2023-08-26T14:02:18.673+0000 7fb612674700 -1 *** Caught signal (Aborted) ** 2023-08-26T14:02:18.695 INFO:tasks.ceph.cluster2.osd.0.smithi124.stderr: in thread 7fb612674700 thread_name:bstore_aio 2023-08-26T14:02:18.695 INFO:tasks.ceph.cluster2.osd.0.smithi124.stderr: 2023-08-26T14:02:18.695 INFO:tasks.ceph.cluster2.osd.0.smithi124.stderr: ceph version 18.0.0-5706-ge7eb9853 (e7eb9853fddaf661c68f207346131a7e00d8ed2f) reef (dev) 2023-08-26T14:02:18.696 INFO:tasks.ceph.cluster2.osd.0.smithi124.stderr: 1: /lib64/libpthread.so.0(+0x12ce0) [0x7fb6251a5ce0] 2023-08-26T14:02:18.696 INFO:tasks.ceph.cluster2.osd.0.smithi124.stderr: 2: gsignal() 2023-08-26T14:02:18.696 INFO:tasks.ceph.cluster2.osd.0.smithi124.stderr: 3: abort() 2023-08-26T14:02:18.696 INFO:tasks.ceph.cluster2.osd.0.smithi124.stderr: 4: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string, std::allocator > const&)+0x197) [0x56452487da75] 2023-08-26T14:02:18.696 INFO:tasks.ceph.cluster2.osd.0.smithi124.stderr: 5: (KernelDevice::_aio_thread()+0x1686) [0x5645254a7416] 2023-08-26T14:02:18.697 INFO:tasks.ceph.cluster2.osd.0.smithi124.stderr: 6: (KernelDevice::AioCompletionThread::entry()+0x11) [0x5645254aced1] 2023-08-26T14:02:18.697 INFO:tasks.ceph.cluster2.osd.0.smithi124.stderr: 7: /lib64/libpthread.so.0(+0x81cf) [0x7fb62519b1cf] 2023-08-26T14:02:18.697 INFO:tasks.ceph.cluster2.osd.0.smithi124.stderr: 8: clone() 2023-08-26T14:02:18.697 INFO:tasks.ceph.cluster2.osd.0.smithi124.stderr: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2023-08-26T14:02:18.697 INFO:tasks.ceph.cluster2.osd.0.smithi124.stderr: 2023-08-26T14:02:18.705 INFO:tasks.ceph.cluster2.osd.0.smithi124.stderr: -912> 2023-08-26T14:02:18.669+0000 7fb612674700 -1 bdev(0x56452769a000 /var/lib/ceph/osd/cluster2-0/block) _aio_thread got r=-28 ((28) No space left on device) 2023-08-26T14:02:18.705 INFO:tasks.ceph.cluster2.osd.0.smithi124.stderr: -785> 2023-08-26T14:02:18.671+0000 7fb612674700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/18.0.0-5706-ge7eb9853/rpm/el8/BUILD/ceph-18.0.0-5706-ge7eb9853/src/blk/kernel/KernelDevice.cc: In function 'void KernelDevice::_aio_thread()' thread 7fb612674700 time 2023-08-26T14:02:18.670361+0000 2023-08-26T14:02:18.705 INFO:tasks.ceph.cluster2.osd.0.smithi124.stderr:/home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/18.0.0-5706-ge7eb9853/rpm/el8/BUILD/ceph-18.0.0-5706-ge7eb9853/src/blk/kernel/KernelDevice.cc: 633: ceph_abort_msg("Unexpected IO error. This may suggest a hardware issue. Please check your kernel log!") 2023-08-26T14:02:18.706 INFO:tasks.ceph.cluster2.osd.0.smithi124.stderr: 2023-08-26T14:02:18.706 INFO:tasks.ceph.cluster2.osd.0.smithi124.stderr: ceph version 18.0.0-5706-ge7eb9853 (e7eb9853fddaf661c68f207346131a7e00d8ed2f) reef (dev) 2023-08-26T14:02:18.706 INFO:tasks.ceph.cluster2.osd.0.smithi124.stderr: 1: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string, std::allocator > const&)+0xd7) [0x56452487d9b5] 2023-08-26T14:02:18.706 INFO:tasks.ceph.cluster2.osd.0.smithi124.stderr: 2: (KernelDevice::_aio_thread()+0x1686) [0x5645254a7416] 2023-08-26T14:02:18.706 INFO:tasks.ceph.cluster2.osd.0.smithi124.stderr: 3: (KernelDevice::AioCompletionThread::entry()+0x11) [0x5645254aced1] 2023-08-26T14:02:18.707 INFO:tasks.ceph.cluster2.osd.0.smithi124.stderr: 4: /lib64/libpthread.so.0(+0x81cf) [0x7fb62519b1cf] 2023-08-26T14:02:18.707 INFO:tasks.ceph.cluster2.osd.0.smithi124.stderr: 5: clone() 2023-08-26T14:02:18.707 INFO:tasks.ceph.cluster2.osd.0.smithi124.stderr: 2023-08-26T14:02:18.707 INFO:tasks.ceph.cluster2.osd.0.smithi124.stderr: -656> 2023-08-26T14:02:18.673+0000 7fb612674700 -1 *** Caught signal (Aborted) ** 2023-08-26T14:02:18.707 INFO:tasks.ceph.cluster2.osd.0.smithi124.stderr: in thread 7fb612674700 thread_name:bstore_aio 2023-08-26T14:02:18.708 INFO:tasks.ceph.cluster2.osd.0.smithi124.stderr: 2023-08-26T14:02:18.708 INFO:tasks.ceph.cluster2.osd.0.smithi124.stderr: ceph version 18.0.0-5706-ge7eb9853 (e7eb9853fddaf661c68f207346131a7e00d8ed2f) reef (dev) 2023-08-26T14:02:18.708 INFO:tasks.ceph.cluster2.osd.0.smithi124.stderr: 1: /lib64/libpthread.so.0(+0x12ce0) [0x7fb6251a5ce0] 2023-08-26T14:02:18.708 INFO:tasks.ceph.cluster2.osd.0.smithi124.stderr: 2: gsignal() 2023-08-26T14:02:18.708 INFO:tasks.ceph.cluster2.osd.0.smithi124.stderr: 3: abort() 2023-08-26T14:02:18.708 INFO:tasks.ceph.cluster2.osd.0.smithi124.stderr: 4: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string, std::allocator > const&)+0x197) [0x56452487da75] 2023-08-26T14:02:18.709 INFO:tasks.ceph.cluster2.osd.0.smithi124.stderr: 5: (KernelDevice::_aio_thread()+0x1686) [0x5645254a7416] 2023-08-26T14:02:18.709 INFO:tasks.ceph.cluster2.osd.0.smithi124.stderr: 6: (KernelDevice::AioCompletionThread::entry()+0x11) [0x5645254aced1] 2023-08-26T14:02:18.709 INFO:tasks.ceph.cluster2.osd.0.smithi124.stderr: 7: /lib64/libpthread.so.0(+0x81cf) [0x7fb62519b1cf] 2023-08-26T14:02:18.709 INFO:tasks.ceph.cluster2.osd.0.smithi124.stderr: 8: clone() 2023-08-26T14:02:18.709 INFO:tasks.ceph.cluster2.osd.0.smithi124.stderr: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2023-08-26T14:02:18.710 INFO:tasks.ceph.cluster2.osd.0.smithi124.stderr: 2023-08-26T14:02:18.911 INFO:tasks.ceph.cluster2.osd.0.smithi124.stderr:daemon-helper: command failed with exit status 5 2023-08-26T14:02:20.398 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:1400 IOs, 505335808 bytes 2023-08-26T14:02:20.482 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:1500 IOs, 505956352 bytes 2023-08-26T14:02:20.583 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:1600 IOs, 518523904 bytes 2023-08-26T14:02:20.754 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:1700 IOs, 597000192 bytes 2023-08-26T14:02:20.904 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:1800 IOs, 656526336 bytes 2023-08-26T14:02:21.075 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:1900 IOs, 723594240 bytes 2023-08-26T14:02:21.272 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:2000 IOs, 807491584 bytes 2023-08-26T14:02:21.353 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:2100 IOs, 812390400 bytes 2023-08-26T14:02:21.588 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:2200 IOs, 930173952 bytes 2023-08-26T14:02:21.801 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:2300 IOs, 1012279296 bytes 2023-08-26T14:02:21.940 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:2400 IOs, 1051709440 bytes 2023-08-26T14:02:22.034 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:2500 IOs, 1090204672 bytes 2023-08-26T14:02:22.162 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:2600 IOs, 1111663616 bytes 2023-08-26T14:02:22.277 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:2700 IOs, 1142232064 bytes 2023-08-26T14:02:22.328 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.0 is failed for ~0s 2023-08-26T14:02:22.389 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:2800 IOs, 1175224320 bytes 2023-08-26T14:02:22.513 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:2900 IOs, 1200433152 bytes 2023-08-26T14:02:22.684 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:3000 IOs, 1273192448 bytes 2023-08-26T14:02:22.782 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:3100 IOs, 1285283840 bytes 2023-08-26T14:02:22.900 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:3200 IOs, 1313845248 bytes 2023-08-26T14:02:23.029 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:3300 IOs, 1346435072 bytes 2023-08-26T14:02:23.157 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:3400 IOs, 1387735040 bytes 2023-08-26T14:02:23.254 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:3500 IOs, 1399796736 bytes 2023-08-26T14:02:23.341 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:3600 IOs, 1410105344 bytes 2023-08-26T14:02:23.448 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:3700 IOs, 1439766528 bytes 2023-08-26T14:02:23.549 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:3800 IOs, 1460466688 bytes 2023-08-26T14:02:23.653 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:3900 IOs, 1475015680 bytes 2023-08-26T14:02:23.830 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:4000 IOs, 1576335360 bytes 2023-08-26T14:02:24.029 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:4100 IOs, 1666217984 bytes 2023-08-26T14:02:24.113 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:4200 IOs, 1674452992 bytes 2023-08-26T14:02:24.216 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:4300 IOs, 1699176448 bytes 2023-08-26T14:02:24.322 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:4400 IOs, 1742477312 bytes 2023-08-26T14:02:24.534 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:4500 IOs, 1832474624 bytes 2023-08-26T14:02:24.650 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:4600 IOs, 1861053440 bytes 2023-08-26T14:02:24.761 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:4700 IOs, 1882980352 bytes 2023-08-26T14:02:24.917 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:4800 IOs, 1946672128 bytes 2023-08-26T14:02:25.076 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:4900 IOs, 1999250432 bytes 2023-08-26T14:02:25.324 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:5000 IOs, 2126004224 bytes 2023-08-26T14:02:25.403 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:5100 IOs, 2131788800 bytes 2023-08-26T14:02:25.522 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:5200 IOs, 2171953152 bytes 2023-08-26T14:02:25.684 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:5300 IOs, 2234912768 bytes 2023-08-26T14:02:25.835 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:5400 IOs, 2294995968 bytes 2023-08-26T14:02:26.036 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:5500 IOs, 2394971136 bytes 2023-08-26T14:02:26.154 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:5600 IOs, 2414722048 bytes 2023-08-26T14:02:26.302 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:5700 IOs, 2472550400 bytes 2023-08-26T14:02:26.395 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:5800 IOs, 2480091136 bytes 2023-08-26T14:02:26.558 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:5900 IOs, 2572468224 bytes 2023-08-26T14:02:26.654 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:6000 IOs, 2598109184 bytes 2023-08-26T14:02:26.789 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:6100 IOs, 2635810816 bytes 2023-08-26T14:02:26.972 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:6200 IOs, 2715416576 bytes 2023-08-26T14:02:27.073 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:6300 IOs, 2737664000 bytes 2023-08-26T14:02:27.219 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:6400 IOs, 2787616768 bytes 2023-08-26T14:02:27.322 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:6500 IOs, 2806057984 bytes 2023-08-26T14:02:27.478 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:6600 IOs, 2858353664 bytes 2023-08-26T14:02:27.611 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:6700 IOs, 2900109312 bytes 2023-08-26T14:02:27.708 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:6800 IOs, 2914879488 bytes 2023-08-26T14:02:27.733 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.0 is failed for ~5s 2023-08-26T14:02:27.841 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:6900 IOs, 2980476928 bytes 2023-08-26T14:02:28.009 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:7000 IOs, 3032842240 bytes 2023-08-26T14:02:28.110 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:7100 IOs, 3064662016 bytes 2023-08-26T14:02:28.290 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:7200 IOs, 3127769088 bytes 2023-08-26T14:02:28.374 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:7300 IOs, 3135572992 bytes 2023-08-26T14:02:28.503 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:7400 IOs, 3176100864 bytes 2023-08-26T14:02:28.653 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:7500 IOs, 3231224832 bytes 2023-08-26T14:02:28.813 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:7600 IOs, 3292697600 bytes 2023-08-26T14:02:28.972 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:7700 IOs, 3372969984 bytes 2023-08-26T14:02:29.120 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:7800 IOs, 3400964096 bytes 2023-08-26T14:02:29.244 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:7900 IOs, 3457038336 bytes 2023-08-26T14:02:29.330 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ error_code=124 2023-08-26T14:02:29.331 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ set -e 2023-08-26T14:02:29.331 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ '[' 124 -eq 124 ']' 2023-08-26T14:02:29.331 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ return 0 2023-08-26T14:02:29.331 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ wait_for_image_present cluster1 mirror image_2 present 2023-08-26T14:02:29.332 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster1 2023-08-26T14:02:29.332 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T14:02:29.332 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=image_2 2023-08-26T14:02:29.332 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local state=present 2023-08-26T14:02:29.332 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image_id= 2023-08-26T14:02:29.333 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local s 2023-08-26T14:02:29.333 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ test -n '' 2023-08-26T14:02:29.333 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ get_image_id cluster1 mirror image_2 2023-08-26T14:02:29.333 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local cluster=cluster1 2023-08-26T14:02:29.333 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local pool=mirror 2023-08-26T14:02:29.333 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local image=image_2 2023-08-26T14:02:29.334 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ rbd --cluster=cluster1 info mirror/image_2 2023-08-26T14:02:29.334 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2023-08-26T14:02:29.374 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ image_id=12c01feb687f 2023-08-26T14:02:29.375 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2023-08-26T14:02:29.375 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ sleep 0.1 2023-08-26T14:02:29.476 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ test_image_present cluster1 mirror image_2 present 12c01feb687f 2023-08-26T14:02:29.476 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster1 2023-08-26T14:02:29.477 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T14:02:29.477 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=image_2 2023-08-26T14:02:29.477 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local test_state=present 2023-08-26T14:02:29.477 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image_id=12c01feb687f 2023-08-26T14:02:29.477 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local current_state=deleted 2023-08-26T14:02:29.478 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local current_image_id 2023-08-26T14:02:29.478 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ get_image_id cluster1 mirror image_2 2023-08-26T14:02:29.478 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local cluster=cluster1 2023-08-26T14:02:29.478 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local pool=mirror 2023-08-26T14:02:29.478 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local image=image_2 2023-08-26T14:02:29.479 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ rbd --cluster=cluster1 info mirror/image_2 2023-08-26T14:02:29.479 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2023-08-26T14:02:29.520 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ current_image_id=12c01feb687f 2023-08-26T14:02:29.520 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ test -n 12c01feb687f 2023-08-26T14:02:29.520 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ test -z 12c01feb687f -o 12c01feb687f = 12c01feb687f 2023-08-26T14:02:29.520 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ current_state=present 2023-08-26T14:02:29.521 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ test present = present 2023-08-26T14:02:29.521 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ return 0 2023-08-26T14:02:29.521 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ disable_mirror cluster2 mirror image_2 2023-08-26T14:02:29.521 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster2 2023-08-26T14:02:29.521 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T14:02:29.522 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=image_2 2023-08-26T14:02:29.522 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rbd --cluster=cluster2 mirror image disable mirror/image_2 2023-08-26T14:02:30.358 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi124.stderr:2023-08-26T14:02:30.356+0000 7f0338336700 -1 rbd::mirror::image_replayer::snapshot::Replayer: 0x557990095c00 scan_remote_mirror_snapshots: failed to locate remote start snapshot: snap_id=370 2023-08-26T14:02:30.359 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi124.stderr:2023-08-26T14:02:30.356+0000 7f0338336700 -1 rbd::mirror::image_replayer::snapshot::Replayer: 0x557990095c00 scan_remote_mirror_snapshots: split-brain detected: failed to find matching non-primary snapshot in remote image: local_snap_id_start=370, local_snap_ns=[mirror state=non-primary, complete=1, mirror_peer_uuids=, primary_mirror_uuid=d88cafc5-6d65-4864-9230-fe3ef515f83d, primary_snap_id=172, last_copied_object_number=128, snap_seqs={370=18446744073709551614}] 2023-08-26T14:02:33.137 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.0 is failed for ~11s 2023-08-26T14:02:35.358 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:Mirroring disabled 2023-08-26T14:02:35.366 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ wait_for_image_present cluster1 mirror image_2 deleted 2023-08-26T14:02:35.367 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster1 2023-08-26T14:02:35.367 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T14:02:35.367 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=image_2 2023-08-26T14:02:35.367 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local state=deleted 2023-08-26T14:02:35.367 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image_id= 2023-08-26T14:02:35.368 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local s 2023-08-26T14:02:35.368 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ test -n '' 2023-08-26T14:02:35.368 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ get_image_id cluster1 mirror image_2 2023-08-26T14:02:35.368 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local cluster=cluster1 2023-08-26T14:02:35.368 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local pool=mirror 2023-08-26T14:02:35.369 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local image=image_2 2023-08-26T14:02:35.369 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ rbd --cluster=cluster1 info mirror/image_2 2023-08-26T14:02:35.369 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2023-08-26T14:02:35.406 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:rbd: error opening image image_2: (2) No such file or directory 2023-08-26T14:02:35.410 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ image_id= 2023-08-26T14:02:35.410 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2023-08-26T14:02:35.410 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ sleep 0.1 2023-08-26T14:02:35.512 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ test_image_present cluster1 mirror image_2 deleted '' 2023-08-26T14:02:35.513 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster1 2023-08-26T14:02:35.513 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T14:02:35.513 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=image_2 2023-08-26T14:02:35.513 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local test_state=deleted 2023-08-26T14:02:35.513 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image_id= 2023-08-26T14:02:35.514 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local current_state=deleted 2023-08-26T14:02:35.514 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local current_image_id 2023-08-26T14:02:35.514 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ get_image_id cluster1 mirror image_2 2023-08-26T14:02:35.514 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local cluster=cluster1 2023-08-26T14:02:35.514 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local pool=mirror 2023-08-26T14:02:35.514 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local image=image_2 2023-08-26T14:02:35.515 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ rbd --cluster=cluster1 info mirror/image_2 2023-08-26T14:02:35.515 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2023-08-26T14:02:35.548 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:rbd: error opening image image_2: (2) No such file or directory 2023-08-26T14:02:35.552 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ current_image_id= 2023-08-26T14:02:35.552 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ test -n '' 2023-08-26T14:02:35.552 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ test deleted = deleted 2023-08-26T14:02:35.552 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ return 0 2023-08-26T14:02:35.553 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ purge_snapshots cluster2 mirror image_2 2023-08-26T14:02:35.553 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster2 2023-08-26T14:02:35.553 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T14:02:35.553 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=image_2 2023-08-26T14:02:35.553 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rbd --cluster cluster2 snap purge mirror/image_2 2023-08-26T14:02:35.596 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ remove_image_retry cluster2 mirror image_2 2023-08-26T14:02:35.597 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster2 2023-08-26T14:02:35.597 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T14:02:35.597 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=image_2 2023-08-26T14:02:35.597 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ for s in 0 1 2 4 8 16 32 2023-08-26T14:02:35.597 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ sleep 0 2023-08-26T14:02:35.598 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ remove_image cluster2 mirror image_2 2023-08-26T14:02:35.598 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster2 2023-08-26T14:02:35.599 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T14:02:35.599 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=image_2 2023-08-26T14:02:35.599 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_2 2023-08-26T14:02:35.641 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rbd --cluster=cluster2 rm mirror/image_2 2023-08-26T14:02:38.541 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.0 is failed for ~16s 2023-08-26T14:02:43.945 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.0 is failed for ~22s 2023-08-26T14:02:49.348 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.0 is failed for ~27s 2023-08-26T14:02:54.752 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.0 is failed for ~32s 2023-08-26T14:02:57.545 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr: Removing image: 1% complete... Removing image: 2% complete... Removing image: 3% complete... Removing image: 4% complete... Removing image: 5% complete... Removing image: 6% complete... Removing image: 7% complete... Removing image: 8% complete... Removing image: 9% complete... Removing image: 10% complete... Removing image: 11% complete... Removing image: 12% complete... Removing image: 13% complete... Removing image: 14% complete... Removing image: 15% complete... Removing image: 16% complete... Removing image: 17% complete... Removing image: 18% complete... Removing image: 19% complete... Removing image: 20% complete... Removing image: 21% complete... Removing image: 22% complete... Removing image: 23% complete... Removing image: 24% complete... Removing image: 25% complete... Removing image: 26% complete... Removing image: 27% complete... Removing image: 28% complete... Removing image: 29% complete... Removing image: 30% complete... Removing image: 31% complete... Removing image: 32% complete... Removing image: 33% complete... Removing image: 34% complete... Removing image: 35% complete... Removing image: 36% complete... Removing image: 37% complete... Removing image: 38% complete... Removing image: 39% complete... Removing image: 40% complete... Removing image: 41% complete... Removing image: 42% complete... Removing image: 43% complete... Removing image: 44% complete... Removing image: 45% complete... Removing image: 46% complete... Removing image: 47% complete... Removing image: 48% complete... Removing image: 49% complete... Removing image: 50% complete... Removing image: 51% complete... Removing image: 52% complete... Removing image: 53% complete... Removing image: 54% complete... Removing image: 55% complete... Removing image: 56% complete... Removing image: 57% complete... Removing image: 58% complete... Removing image: 59% complete... Removing image: 60% complete... Removing image: 61% complete... Removing image: 62% complete... Removing image: 63% complete... Removing image: 64% complete... Removing image: 65% complete... Removing image: 66% complete... Removing image: 67% complete... Removing image: 68% complete... Removing image: 69% complete... Removing image: 70% complete... Removing image: 71% complete... Removing image: 72% complete... Removing image: 73% complete... Removing image: 74% complete... Removing image: 75% complete... Removing image: 76% complete... Removing image: 77% complete... Removing image: 78% complete... Removing image: 79% complete... Removing image: 80% complete... Removing image: 81% complete... Removing image: 82% complete... Removing image: 83% complete... Removing image: 84% complete... Removing image: 85% complete... Removing image: 86% complete... Removing image: 87% complete... Removing image: 88% complete... Removing image: 89% complete... Removing image: 90% complete... Removing image: 91% complete... Removing image: 92% complete... Removing image: 93% complete... Removing image: 94% complete... Removing image: 95% complete... Removing image: 96% complete... Removing image: 97% complete... Removing image: 98% complete... Removing image: 99% complete... Removing image: 100% complete...done. 2023-08-26T14:02:57.549 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ return 0 2023-08-26T14:02:57.550 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ for i in `seq 1 10` 2023-08-26T14:02:57.550 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ image=image_3 2023-08-26T14:02:57.550 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ create_image cluster2 mirror image_3 512M 2023-08-26T14:02:57.550 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster2 2023-08-26T14:02:57.551 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ shift 2023-08-26T14:02:57.551 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T14:02:57.551 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ shift 2023-08-26T14:02:57.551 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=image_3 2023-08-26T14:02:57.551 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ shift 2023-08-26T14:02:57.552 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local size=128 2023-08-26T14:02:57.552 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ '[' -n 512M ']' 2023-08-26T14:02:57.552 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ size=512M 2023-08-26T14:02:57.552 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ shift 2023-08-26T14:02:57.553 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rbd --cluster cluster2 create --size 512M --image-feature layering,exclusive-lock mirror/image_3 2023-08-26T14:02:57.596 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ enable_mirror cluster2 mirror image_3 2023-08-26T14:02:57.596 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster2 2023-08-26T14:02:57.596 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T14:02:57.597 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=image_3 2023-08-26T14:02:57.597 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local mode=snapshot 2023-08-26T14:02:57.597 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_3 snapshot 2023-08-26T14:02:58.382 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:Mirroring enabled 2023-08-26T14:02:58.389 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rbd --cluster=cluster2 info mirror/image_3 2023-08-26T14:02:58.426 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:rbd image 'image_3': 2023-08-26T14:02:58.427 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: size 512 MiB in 128 objects 2023-08-26T14:02:58.427 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: order 22 (4 MiB objects) 2023-08-26T14:02:58.427 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: snapshot_count: 1 2023-08-26T14:02:58.427 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: id: 18928050a7d9 2023-08-26T14:02:58.427 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: block_name_prefix: rbd_data.18928050a7d9 2023-08-26T14:02:58.428 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: format: 2 2023-08-26T14:02:58.428 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: features: layering, exclusive-lock 2023-08-26T14:02:58.428 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: op_features: 2023-08-26T14:02:58.428 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: flags: 2023-08-26T14:02:58.428 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: create_timestamp: Sat Aug 26 14:02:57 2023 2023-08-26T14:02:58.429 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: access_timestamp: Sat Aug 26 14:02:57 2023 2023-08-26T14:02:58.429 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: modify_timestamp: Sat Aug 26 14:02:57 2023 2023-08-26T14:02:58.429 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: mirroring state: enabled 2023-08-26T14:02:58.429 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: mirroring mode: snapshot 2023-08-26T14:02:58.429 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: mirroring global id: 154ea53f-3240-4090-b89e-ee1dedb22220 2023-08-26T14:02:58.430 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: mirroring primary: true 2023-08-26T14:02:58.430 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ stress_write_image cluster2 mirror image_3 2023-08-26T14:02:58.431 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster2 2023-08-26T14:02:58.431 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T14:02:58.431 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=image_3 2023-08-26T14:02:58.431 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2023-08-26T14:02:58.432 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local duration=12 2023-08-26T14:02:58.432 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ set +e 2023-08-26T14:02:58.433 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ timeout 12s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_3 --debug-rbd=20 --debug-journaler=20 2023-08-26T14:02:58.542 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:100 IOs, 609280 bytes 2023-08-26T14:02:58.622 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:200 IOs, 12365824 bytes 2023-08-26T14:02:58.790 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:300 IOs, 95516672 bytes 2023-08-26T14:02:58.877 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:400 IOs, 104549376 bytes 2023-08-26T14:02:59.026 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:500 IOs, 158877696 bytes 2023-08-26T14:02:59.138 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:600 IOs, 186375168 bytes 2023-08-26T14:02:59.280 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:700 IOs, 244633600 bytes 2023-08-26T14:02:59.351 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:800 IOs, 245240832 bytes 2023-08-26T14:02:59.507 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:900 IOs, 327608320 bytes 2023-08-26T14:02:59.617 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:1000 IOs, 356063232 bytes 2023-08-26T14:02:59.717 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:1100 IOs, 382678016 bytes 2023-08-26T14:02:59.798 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:1200 IOs, 390133760 bytes 2023-08-26T14:02:59.963 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:1300 IOs, 460617728 bytes 2023-08-26T14:03:00.094 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:1400 IOs, 518646784 bytes 2023-08-26T14:03:00.155 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.0 is failed for ~38s 2023-08-26T14:03:00.208 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:1500 IOs, 534149120 bytes 2023-08-26T14:03:00.335 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:1600 IOs, 567441408 bytes 2023-08-26T14:03:00.462 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:1700 IOs, 615165952 bytes 2023-08-26T14:03:00.640 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:1800 IOs, 681539584 bytes 2023-08-26T14:03:00.835 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:1900 IOs, 780067840 bytes 2023-08-26T14:03:00.995 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:2000 IOs, 855396352 bytes 2023-08-26T14:03:01.102 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:2100 IOs, 880216064 bytes 2023-08-26T14:03:01.225 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:2200 IOs, 923608064 bytes 2023-08-26T14:03:01.316 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:2300 IOs, 938996736 bytes 2023-08-26T14:03:01.452 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:2400 IOs, 975407104 bytes 2023-08-26T14:03:01.565 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:2500 IOs, 1011335168 bytes 2023-08-26T14:03:01.660 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:2600 IOs, 1028950016 bytes 2023-08-26T14:03:01.784 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:2700 IOs, 1075300352 bytes 2023-08-26T14:03:01.925 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:2800 IOs, 1117143040 bytes 2023-08-26T14:03:02.033 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:2900 IOs, 1159380992 bytes 2023-08-26T14:03:02.139 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:3000 IOs, 1182515200 bytes 2023-08-26T14:03:02.325 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:3100 IOs, 1254512640 bytes 2023-08-26T14:03:02.456 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:3200 IOs, 1294046208 bytes 2023-08-26T14:03:02.624 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:3300 IOs, 1361707008 bytes 2023-08-26T14:03:02.772 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:3400 IOs, 1419965440 bytes 2023-08-26T14:03:02.906 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:3500 IOs, 1473702912 bytes 2023-08-26T14:03:03.103 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:3600 IOs, 1557097472 bytes 2023-08-26T14:03:03.312 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:3700 IOs, 1601595392 bytes 2023-08-26T14:03:03.440 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:3800 IOs, 1642539008 bytes 2023-08-26T14:03:03.530 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:3900 IOs, 1652794368 bytes 2023-08-26T14:03:03.640 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:4000 IOs, 1708142592 bytes 2023-08-26T14:03:03.755 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:4100 IOs, 1712299008 bytes 2023-08-26T14:03:03.866 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:4200 IOs, 1747325952 bytes 2023-08-26T14:03:03.984 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:4300 IOs, 1784102912 bytes 2023-08-26T14:03:04.068 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:4400 IOs, 1785821184 bytes 2023-08-26T14:03:04.191 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:4500 IOs, 1817797632 bytes 2023-08-26T14:03:04.358 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:4600 IOs, 1883272192 bytes 2023-08-26T14:03:04.507 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:4700 IOs, 1942593536 bytes 2023-08-26T14:03:04.705 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:4800 IOs, 2035793920 bytes 2023-08-26T14:03:04.872 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:4900 IOs, 2123218944 bytes 2023-08-26T14:03:05.017 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:5000 IOs, 2157617152 bytes 2023-08-26T14:03:05.152 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:5100 IOs, 2205897728 bytes 2023-08-26T14:03:05.318 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:5200 IOs, 2279993344 bytes 2023-08-26T14:03:05.457 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:5300 IOs, 2325443584 bytes 2023-08-26T14:03:05.558 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.0 is failed for ~43s 2023-08-26T14:03:05.571 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:5400 IOs, 2353289216 bytes 2023-08-26T14:03:05.658 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:5500 IOs, 2371310592 bytes 2023-08-26T14:03:05.774 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:5600 IOs, 2398760960 bytes 2023-08-26T14:03:05.876 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:5700 IOs, 2419323904 bytes 2023-08-26T14:03:05.961 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:5800 IOs, 2426563584 bytes 2023-08-26T14:03:06.030 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:5900 IOs, 2427168768 bytes 2023-08-26T14:03:06.115 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:6000 IOs, 2446576640 bytes 2023-08-26T14:03:06.279 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:6100 IOs, 2515025920 bytes 2023-08-26T14:03:06.405 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:6200 IOs, 2570255360 bytes 2023-08-26T14:03:06.504 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:6300 IOs, 2591975424 bytes 2023-08-26T14:03:06.633 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:6400 IOs, 2635796480 bytes 2023-08-26T14:03:06.747 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:6500 IOs, 2679358464 bytes 2023-08-26T14:03:06.775 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.0 with skip (0.2967012631556972) > weight (0.25) 2023-08-26T14:03:06.775 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.1 with skip (0.4034605142357386) > weight (0.25) 2023-08-26T14:03:06.775 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.2 with skip (0.5444374375806122) > weight (0.25) 2023-08-26T14:03:06.776 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.3 with skip (0.6095127036221446) > weight (0.25) 2023-08-26T14:03:06.776 INFO:tasks.rbd_mirror_thrash:waiting for 103 secs before thrashing 2023-08-26T14:03:06.932 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:6600 IOs, 2748331008 bytes 2023-08-26T14:03:07.053 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:6700 IOs, 2782003200 bytes 2023-08-26T14:03:07.184 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:6800 IOs, 2815772672 bytes 2023-08-26T14:03:07.301 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:6900 IOs, 2852025344 bytes 2023-08-26T14:03:07.398 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:7000 IOs, 2863463424 bytes 2023-08-26T14:03:07.562 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:7100 IOs, 2926435328 bytes 2023-08-26T14:03:07.638 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:7200 IOs, 2928941056 bytes 2023-08-26T14:03:07.717 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:7300 IOs, 2934475776 bytes 2023-08-26T14:03:07.864 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:7400 IOs, 2998110208 bytes 2023-08-26T14:03:07.975 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:7500 IOs, 3020570624 bytes 2023-08-26T14:03:08.105 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:7600 IOs, 3073090560 bytes 2023-08-26T14:03:08.284 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:7700 IOs, 3168196608 bytes 2023-08-26T14:03:08.479 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:7800 IOs, 3243854848 bytes 2023-08-26T14:03:08.607 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:7900 IOs, 3284204544 bytes 2023-08-26T14:03:08.757 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:8000 IOs, 3343113216 bytes 2023-08-26T14:03:08.850 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:8100 IOs, 3363553280 bytes 2023-08-26T14:03:08.965 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:8200 IOs, 3390097408 bytes 2023-08-26T14:03:09.123 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:8300 IOs, 3470155776 bytes 2023-08-26T14:03:09.249 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:8400 IOs, 3499291648 bytes 2023-08-26T14:03:09.416 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:8500 IOs, 3588020224 bytes 2023-08-26T14:03:09.520 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:8600 IOs, 3615416320 bytes 2023-08-26T14:03:09.618 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:8700 IOs, 3616039936 bytes 2023-08-26T14:03:09.696 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:8800 IOs, 3616651264 bytes 2023-08-26T14:03:09.807 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:8900 IOs, 3646208000 bytes 2023-08-26T14:03:09.938 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:9000 IOs, 3694263296 bytes 2023-08-26T14:03:10.048 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:9100 IOs, 3753065472 bytes 2023-08-26T14:03:10.189 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:9200 IOs, 3791783936 bytes 2023-08-26T14:03:10.282 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:9300 IOs, 3792402432 bytes 2023-08-26T14:03:10.394 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:9400 IOs, 3819528192 bytes 2023-08-26T14:03:10.437 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ error_code=124 2023-08-26T14:03:10.437 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ set -e 2023-08-26T14:03:10.438 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ '[' 124 -eq 124 ']' 2023-08-26T14:03:10.438 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ return 0 2023-08-26T14:03:10.438 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ wait_for_image_present cluster1 mirror image_3 present 2023-08-26T14:03:10.438 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster1 2023-08-26T14:03:10.439 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T14:03:10.439 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=image_3 2023-08-26T14:03:10.439 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local state=present 2023-08-26T14:03:10.439 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image_id= 2023-08-26T14:03:10.439 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local s 2023-08-26T14:03:10.440 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ test -n '' 2023-08-26T14:03:10.440 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ get_image_id cluster1 mirror image_3 2023-08-26T14:03:10.440 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local cluster=cluster1 2023-08-26T14:03:10.441 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local pool=mirror 2023-08-26T14:03:10.441 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local image=image_3 2023-08-26T14:03:10.441 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ rbd --cluster=cluster1 info mirror/image_3 2023-08-26T14:03:10.442 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2023-08-26T14:03:10.481 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ image_id=12c0114a5a09 2023-08-26T14:03:10.481 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2023-08-26T14:03:10.481 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ sleep 0.1 2023-08-26T14:03:10.582 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ test_image_present cluster1 mirror image_3 present 12c0114a5a09 2023-08-26T14:03:10.583 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster1 2023-08-26T14:03:10.583 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T14:03:10.583 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=image_3 2023-08-26T14:03:10.583 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local test_state=present 2023-08-26T14:03:10.584 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image_id=12c0114a5a09 2023-08-26T14:03:10.584 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local current_state=deleted 2023-08-26T14:03:10.584 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local current_image_id 2023-08-26T14:03:10.584 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ get_image_id cluster1 mirror image_3 2023-08-26T14:03:10.584 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local cluster=cluster1 2023-08-26T14:03:10.585 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local pool=mirror 2023-08-26T14:03:10.585 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local image=image_3 2023-08-26T14:03:10.585 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ rbd --cluster=cluster1 info mirror/image_3 2023-08-26T14:03:10.585 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2023-08-26T14:03:10.625 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ current_image_id=12c0114a5a09 2023-08-26T14:03:10.625 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ test -n 12c0114a5a09 2023-08-26T14:03:10.626 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ test -z 12c0114a5a09 -o 12c0114a5a09 = 12c0114a5a09 2023-08-26T14:03:10.626 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ current_state=present 2023-08-26T14:03:10.626 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ test present = present 2023-08-26T14:03:10.626 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ return 0 2023-08-26T14:03:10.626 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ disable_mirror cluster2 mirror image_3 2023-08-26T14:03:10.627 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster2 2023-08-26T14:03:10.627 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T14:03:10.627 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=image_3 2023-08-26T14:03:10.627 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rbd --cluster=cluster2 mirror image disable mirror/image_3 2023-08-26T14:03:10.962 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.0 is failed for ~49s 2023-08-26T14:03:11.387 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi124.stderr:2023-08-26T14:03:11.386+0000 7f0338b37700 -1 rbd::mirror::image_replayer::snapshot::Replayer: 0x55798c784400 scan_remote_mirror_snapshots: failed to locate remote start snapshot: snap_id=372 2023-08-26T14:03:11.388 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi124.stderr:2023-08-26T14:03:11.386+0000 7f0338b37700 -1 rbd::mirror::image_replayer::snapshot::Replayer: 0x55798c784400 scan_remote_mirror_snapshots: split-brain detected: failed to find matching non-primary snapshot in remote image: local_snap_id_start=372, local_snap_ns=[mirror state=non-primary, complete=1, mirror_peer_uuids=, primary_mirror_uuid=d88cafc5-6d65-4864-9230-fe3ef515f83d, primary_snap_id=174, last_copied_object_number=128, snap_seqs={372=18446744073709551614}] 2023-08-26T14:03:16.366 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.0 is failed for ~54s 2023-08-26T14:03:16.387 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:Mirroring disabled 2023-08-26T14:03:16.395 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ wait_for_image_present cluster1 mirror image_3 deleted 2023-08-26T14:03:16.395 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster1 2023-08-26T14:03:16.395 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T14:03:16.396 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=image_3 2023-08-26T14:03:16.396 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local state=deleted 2023-08-26T14:03:16.396 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image_id= 2023-08-26T14:03:16.396 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local s 2023-08-26T14:03:16.396 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ test -n '' 2023-08-26T14:03:16.397 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ get_image_id cluster1 mirror image_3 2023-08-26T14:03:16.397 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local cluster=cluster1 2023-08-26T14:03:16.397 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local pool=mirror 2023-08-26T14:03:16.397 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local image=image_3 2023-08-26T14:03:16.397 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ rbd --cluster=cluster1 info mirror/image_3 2023-08-26T14:03:16.397 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2023-08-26T14:03:16.434 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:rbd: error opening image image_3: (2) No such file or directory 2023-08-26T14:03:16.438 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ image_id= 2023-08-26T14:03:16.438 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2023-08-26T14:03:16.439 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ sleep 0.1 2023-08-26T14:03:16.540 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ test_image_present cluster1 mirror image_3 deleted '' 2023-08-26T14:03:16.541 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster1 2023-08-26T14:03:16.541 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T14:03:16.541 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=image_3 2023-08-26T14:03:16.541 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local test_state=deleted 2023-08-26T14:03:16.541 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image_id= 2023-08-26T14:03:16.542 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local current_state=deleted 2023-08-26T14:03:16.542 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local current_image_id 2023-08-26T14:03:16.542 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ get_image_id cluster1 mirror image_3 2023-08-26T14:03:16.542 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local cluster=cluster1 2023-08-26T14:03:16.542 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local pool=mirror 2023-08-26T14:03:16.543 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local image=image_3 2023-08-26T14:03:16.543 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ rbd --cluster=cluster1 info mirror/image_3 2023-08-26T14:03:16.543 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2023-08-26T14:03:16.581 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:rbd: error opening image image_3: (2) No such file or directory 2023-08-26T14:03:16.585 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ current_image_id= 2023-08-26T14:03:16.585 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ test -n '' 2023-08-26T14:03:16.585 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ test deleted = deleted 2023-08-26T14:03:16.586 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ return 0 2023-08-26T14:03:16.586 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ purge_snapshots cluster2 mirror image_3 2023-08-26T14:03:16.586 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster2 2023-08-26T14:03:16.586 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T14:03:16.586 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=image_3 2023-08-26T14:03:16.587 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rbd --cluster cluster2 snap purge mirror/image_3 2023-08-26T14:03:16.632 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ remove_image_retry cluster2 mirror image_3 2023-08-26T14:03:16.632 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster2 2023-08-26T14:03:16.632 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T14:03:16.632 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=image_3 2023-08-26T14:03:16.633 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ for s in 0 1 2 4 8 16 32 2023-08-26T14:03:16.633 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ sleep 0 2023-08-26T14:03:16.634 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ remove_image cluster2 mirror image_3 2023-08-26T14:03:16.634 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster2 2023-08-26T14:03:16.634 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T14:03:16.634 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=image_3 2023-08-26T14:03:16.634 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_3 2023-08-26T14:03:16.676 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rbd --cluster=cluster2 rm mirror/image_3 2023-08-26T14:03:21.770 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.0 is failed for ~59s 2023-08-26T14:03:27.175 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.0 is failed for ~65s 2023-08-26T14:03:30.553 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.0 with skip (0.9051399034642054) > weight (0.25) 2023-08-26T14:03:30.554 INFO:tasks.rbd_mirror_thrash:kill cluster2.client.mirror.1 2023-08-26T14:03:30.554 INFO:tasks.rbd_mirror.cluster2.client.mirror.1:Sent signal 15 2023-08-26T14:03:30.554 INFO:tasks.rbd_mirror_thrash:waiting for 4 secs before reviving daemons 2023-08-26T14:03:30.555 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi124.stderr:2023-08-26T14:03:30.553+0000 7f6c3a648700 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.1 (PID: 111646) UID: 1000 2023-08-26T14:03:32.578 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.0 is failed for ~70s 2023-08-26T14:03:34.556 INFO:tasks.rbd_mirror_thrash:waiting for cluster2.client.mirror.1 2023-08-26T14:03:34.556 INFO:teuthology.orchestra.run:waiting for 600 2023-08-26T14:03:34.556 INFO:tasks.rbd_mirror_thrash:reviving cluster2.client.mirror.1 2023-08-26T14:03:34.557 INFO:tasks.rbd_mirror.cluster2.client.mirror.1:Restarting daemon 2023-08-26T14:03:34.557 DEBUG:teuthology.orchestra.run.smithi124:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.1 2023-08-26T14:03:34.560 INFO:tasks.rbd_mirror.cluster2.client.mirror.1:Started 2023-08-26T14:03:34.560 INFO:tasks.rbd_mirror_thrash:waiting for 80 secs before thrashing 2023-08-26T14:03:37.982 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.0 is failed for ~76s 2023-08-26T14:03:40.547 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr: Removing image: 1% complete... Removing image: 2% complete... Removing image: 3% complete... Removing image: 4% complete... Removing image: 5% complete... Removing image: 6% complete... Removing image: 7% complete... Removing image: 8% complete... Removing image: 9% complete... Removing image: 10% complete... Removing image: 11% complete... Removing image: 12% complete... Removing image: 13% complete... Removing image: 14% complete... Removing image: 15% complete... Removing image: 16% complete... Removing image: 17% complete... Removing image: 18% complete... Removing image: 19% complete... Removing image: 20% complete... Removing image: 21% complete... Removing image: 22% complete... Removing image: 23% complete... Removing image: 24% complete... Removing image: 25% complete... Removing image: 26% complete... Removing image: 27% complete... Removing image: 28% complete... Removing image: 29% complete... Removing image: 30% complete... Removing image: 31% complete... Removing image: 32% complete... Removing image: 33% complete... Removing image: 34% complete... Removing image: 35% complete... Removing image: 36% complete... Removing image: 37% complete... Removing image: 38% complete... Removing image: 39% complete... Removing image: 40% complete... Removing image: 41% complete... Removing image: 42% complete... Removing image: 43% complete... Removing image: 44% complete... Removing image: 45% complete... Removing image: 46% complete... Removing image: 47% complete... Removing image: 48% complete... Removing image: 49% complete... Removing image: 50% complete... Removing image: 51% complete... Removing image: 52% complete... Removing image: 53% complete... Removing image: 54% complete... Removing image: 55% complete... Removing image: 56% complete... Removing image: 57% complete... Removing image: 58% complete... Removing image: 59% complete... Removing image: 60% complete... Removing image: 61% complete... Removing image: 62% complete... Removing image: 63% complete... Removing image: 64% complete... Removing image: 65% complete... Removing image: 66% complete... Removing image: 67% complete... Removing image: 68% complete... Removing image: 69% complete... Removing image: 70% complete... Removing image: 71% complete... Removing image: 72% complete... Removing image: 73% complete... Removing image: 74% complete... Removing image: 75% complete... Removing image: 76% complete... Removing image: 77% complete... Removing image: 78% complete... Removing image: 79% complete... Removing image: 80% complete... Removing image: 81% complete... Removing image: 82% complete... Removing image: 83% complete... Removing image: 84% complete... Removing image: 85% complete... Removing image: 86% complete... Removing image: 87% complete... Removing image: 88% complete... Removing image: 89% complete... Removing image: 90% complete... Removing image: 91% complete... Removing image: 92% complete... Removing image: 93% complete... Removing image: 94% complete... Removing image: 95% complete... Removing image: 96% complete... Removing image: 97% complete... Removing image: 98% complete... Removing image: 99% complete... Removing image: 100% complete...done. 2023-08-26T14:03:40.552 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ return 0 2023-08-26T14:03:40.552 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ for i in `seq 1 10` 2023-08-26T14:03:40.553 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ image=image_4 2023-08-26T14:03:40.553 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ create_image cluster2 mirror image_4 512M 2023-08-26T14:03:40.553 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster2 2023-08-26T14:03:40.554 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ shift 2023-08-26T14:03:40.554 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T14:03:40.554 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ shift 2023-08-26T14:03:40.555 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=image_4 2023-08-26T14:03:40.555 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ shift 2023-08-26T14:03:40.555 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local size=128 2023-08-26T14:03:40.556 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ '[' -n 512M ']' 2023-08-26T14:03:40.556 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ size=512M 2023-08-26T14:03:40.556 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ shift 2023-08-26T14:03:40.556 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rbd --cluster cluster2 create --size 512M --image-feature layering,exclusive-lock mirror/image_4 2023-08-26T14:03:40.596 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ enable_mirror cluster2 mirror image_4 2023-08-26T14:03:40.596 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster2 2023-08-26T14:03:40.597 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T14:03:40.597 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=image_4 2023-08-26T14:03:40.597 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local mode=snapshot 2023-08-26T14:03:40.598 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_4 snapshot 2023-08-26T14:03:41.400 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:Mirroring enabled 2023-08-26T14:03:41.407 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rbd --cluster=cluster2 info mirror/image_4 2023-08-26T14:03:41.445 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:rbd image 'image_4': 2023-08-26T14:03:41.445 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: size 512 MiB in 128 objects 2023-08-26T14:03:41.446 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: order 22 (4 MiB objects) 2023-08-26T14:03:41.446 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: snapshot_count: 1 2023-08-26T14:03:41.446 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: id: 18b36d5a2d1d 2023-08-26T14:03:41.447 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: block_name_prefix: rbd_data.18b36d5a2d1d 2023-08-26T14:03:41.447 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: format: 2 2023-08-26T14:03:41.447 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: features: layering, exclusive-lock 2023-08-26T14:03:41.448 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: op_features: 2023-08-26T14:03:41.448 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: flags: 2023-08-26T14:03:41.449 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: create_timestamp: Sat Aug 26 14:03:40 2023 2023-08-26T14:03:41.449 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: access_timestamp: Sat Aug 26 14:03:40 2023 2023-08-26T14:03:41.449 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: modify_timestamp: Sat Aug 26 14:03:40 2023 2023-08-26T14:03:41.450 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: mirroring state: enabled 2023-08-26T14:03:41.450 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: mirroring mode: snapshot 2023-08-26T14:03:41.450 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: mirroring global id: 0df27f7e-c961-4a0f-a3f7-f081874bb9b5 2023-08-26T14:03:41.451 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: mirroring primary: true 2023-08-26T14:03:41.452 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ stress_write_image cluster2 mirror image_4 2023-08-26T14:03:41.453 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster2 2023-08-26T14:03:41.453 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T14:03:41.453 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=image_4 2023-08-26T14:03:41.454 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2023-08-26T14:03:41.454 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local duration=13 2023-08-26T14:03:41.454 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ set +e 2023-08-26T14:03:41.455 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ timeout 13s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_4 --debug-rbd=20 --debug-journaler=20 2023-08-26T14:03:41.675 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:100 IOs, 90807296 bytes 2023-08-26T14:03:41.783 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:200 IOs, 120424448 bytes 2023-08-26T14:03:41.921 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:300 IOs, 178199552 bytes 2023-08-26T14:03:42.043 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:400 IOs, 215815168 bytes 2023-08-26T14:03:42.129 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:500 IOs, 239502336 bytes 2023-08-26T14:03:42.276 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:600 IOs, 287266816 bytes 2023-08-26T14:03:42.413 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:700 IOs, 319500288 bytes 2023-08-26T14:03:42.578 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:800 IOs, 411193344 bytes 2023-08-26T14:03:42.737 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:900 IOs, 487598080 bytes 2023-08-26T14:03:42.935 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:1000 IOs, 563878912 bytes 2023-08-26T14:03:43.067 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:1100 IOs, 631070720 bytes 2023-08-26T14:03:43.193 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:1200 IOs, 668526592 bytes 2023-08-26T14:03:43.336 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:1300 IOs, 733425664 bytes 2023-08-26T14:03:43.385 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.0 is failed for ~81s 2023-08-26T14:03:43.520 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:1400 IOs, 803773440 bytes 2023-08-26T14:03:43.657 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:1500 IOs, 848765952 bytes 2023-08-26T14:03:43.791 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:1600 IOs, 890674176 bytes 2023-08-26T14:03:43.960 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:1700 IOs, 952507392 bytes 2023-08-26T14:03:44.045 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:1800 IOs, 968409088 bytes 2023-08-26T14:03:44.197 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:1900 IOs, 1015119872 bytes 2023-08-26T14:03:44.330 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:2000 IOs, 1061599232 bytes 2023-08-26T14:03:44.445 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:2100 IOs, 1095343104 bytes 2023-08-26T14:03:44.778 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:2200 IOs, 1213437952 bytes 2023-08-26T14:03:44.912 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:2300 IOs, 1234944000 bytes 2023-08-26T14:03:45.043 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:2400 IOs, 1274296320 bytes 2023-08-26T14:03:45.151 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:2500 IOs, 1296545792 bytes 2023-08-26T14:03:45.332 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:2600 IOs, 1379329024 bytes 2023-08-26T14:03:45.466 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:2700 IOs, 1427777536 bytes 2023-08-26T14:03:45.541 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:2800 IOs, 1430318080 bytes 2023-08-26T14:03:45.632 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:2900 IOs, 1453809664 bytes 2023-08-26T14:03:45.740 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:3000 IOs, 1483666432 bytes 2023-08-26T14:03:45.968 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:3100 IOs, 1588736000 bytes 2023-08-26T14:03:46.107 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:3200 IOs, 1638755328 bytes 2023-08-26T14:03:46.230 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:3300 IOs, 1668811776 bytes 2023-08-26T14:03:46.407 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:3400 IOs, 1729282048 bytes 2023-08-26T14:03:46.672 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:3500 IOs, 1898734592 bytes 2023-08-26T14:03:46.800 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:3600 IOs, 1967372288 bytes 2023-08-26T14:03:47.010 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:3700 IOs, 2024212480 bytes 2023-08-26T14:03:47.141 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:3800 IOs, 2073414656 bytes 2023-08-26T14:03:47.312 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:3900 IOs, 2139327488 bytes 2023-08-26T14:03:47.469 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:4000 IOs, 2186021888 bytes 2023-08-26T14:03:47.619 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:4100 IOs, 2251112448 bytes 2023-08-26T14:03:47.792 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:4200 IOs, 2325609472 bytes 2023-08-26T14:03:47.914 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:4300 IOs, 2374403072 bytes 2023-08-26T14:03:48.036 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:4400 IOs, 2392932352 bytes 2023-08-26T14:03:48.145 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:4500 IOs, 2423479296 bytes 2023-08-26T14:03:48.314 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:4600 IOs, 2498779136 bytes 2023-08-26T14:03:48.432 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:4700 IOs, 2545617920 bytes 2023-08-26T14:03:48.570 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:4800 IOs, 2580882432 bytes 2023-08-26T14:03:48.690 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:4900 IOs, 2624598016 bytes 2023-08-26T14:03:48.789 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.0 is failed for ~86s 2023-08-26T14:03:48.802 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:5000 IOs, 2652096512 bytes 2023-08-26T14:03:48.943 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:5100 IOs, 2708909056 bytes 2023-08-26T14:03:49.093 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:5200 IOs, 2766786560 bytes 2023-08-26T14:03:49.251 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:5300 IOs, 2833323008 bytes 2023-08-26T14:03:49.381 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:5400 IOs, 2895197184 bytes 2023-08-26T14:03:49.517 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:5500 IOs, 2918781952 bytes 2023-08-26T14:03:49.651 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:5600 IOs, 2961977344 bytes 2023-08-26T14:03:49.834 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:5700 IOs, 3064653824 bytes 2023-08-26T14:03:49.957 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:5800 IOs, 3087975424 bytes 2023-08-26T14:03:50.075 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:5900 IOs, 3148053504 bytes 2023-08-26T14:03:50.209 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:6000 IOs, 3177465856 bytes 2023-08-26T14:03:50.297 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:6100 IOs, 3178074112 bytes 2023-08-26T14:03:50.378 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:6200 IOs, 3178669056 bytes 2023-08-26T14:03:50.463 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:6300 IOs, 3179678720 bytes 2023-08-26T14:03:50.604 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:6400 IOs, 3233471488 bytes 2023-08-26T14:03:50.735 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:6500 IOs, 3280809984 bytes 2023-08-26T14:03:50.865 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:6600 IOs, 3318125568 bytes 2023-08-26T14:03:50.989 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:6700 IOs, 3357745152 bytes 2023-08-26T14:03:51.202 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:6800 IOs, 3457936384 bytes 2023-08-26T14:03:51.311 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:6900 IOs, 3482677248 bytes 2023-08-26T14:03:51.393 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:7000 IOs, 3492011008 bytes 2023-08-26T14:03:51.520 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:7100 IOs, 3535213568 bytes 2023-08-26T14:03:51.593 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:7200 IOs, 3537098752 bytes 2023-08-26T14:03:51.699 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:7300 IOs, 3568429056 bytes 2023-08-26T14:03:51.873 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:7400 IOs, 3639325696 bytes 2023-08-26T14:03:52.051 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:7500 IOs, 3680219136 bytes 2023-08-26T14:03:52.193 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:7600 IOs, 3728575488 bytes 2023-08-26T14:03:52.302 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:7700 IOs, 3757737984 bytes 2023-08-26T14:03:52.441 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:7800 IOs, 3828163584 bytes 2023-08-26T14:03:52.559 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:7900 IOs, 3828783104 bytes 2023-08-26T14:03:52.723 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:8000 IOs, 3887033344 bytes 2023-08-26T14:03:52.854 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:8100 IOs, 3907783680 bytes 2023-08-26T14:03:52.982 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:8200 IOs, 3947737088 bytes 2023-08-26T14:03:53.417 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:8300 IOs, 4004923392 bytes 2023-08-26T14:03:53.518 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:8400 IOs, 4017313792 bytes 2023-08-26T14:03:53.669 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:8500 IOs, 4073018368 bytes 2023-08-26T14:03:53.785 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:8600 IOs, 4100672512 bytes 2023-08-26T14:03:53.903 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:8700 IOs, 4123105280 bytes 2023-08-26T14:03:54.080 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:8800 IOs, 4203888640 bytes 2023-08-26T14:03:54.193 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.0 is failed for ~92s 2023-08-26T14:03:54.267 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:8900 IOs, 4277359616 bytes 2023-08-26T14:03:54.409 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:9000 IOs, 4338889728 bytes 2023-08-26T14:03:54.454 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ error_code=124 2023-08-26T14:03:54.455 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ set -e 2023-08-26T14:03:54.455 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ '[' 124 -eq 124 ']' 2023-08-26T14:03:54.455 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ return 0 2023-08-26T14:03:54.455 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ wait_for_image_present cluster1 mirror image_4 present 2023-08-26T14:03:54.456 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster1 2023-08-26T14:03:54.456 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T14:03:54.456 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=image_4 2023-08-26T14:03:54.457 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local state=present 2023-08-26T14:03:54.457 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image_id= 2023-08-26T14:03:54.457 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local s 2023-08-26T14:03:54.457 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ test -n '' 2023-08-26T14:03:54.458 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ get_image_id cluster1 mirror image_4 2023-08-26T14:03:54.458 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local cluster=cluster1 2023-08-26T14:03:54.458 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local pool=mirror 2023-08-26T14:03:54.458 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local image=image_4 2023-08-26T14:03:54.458 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ rbd --cluster=cluster1 info mirror/image_4 2023-08-26T14:03:54.459 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2023-08-26T14:03:54.498 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ image_id=12c07056e5c1 2023-08-26T14:03:54.498 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2023-08-26T14:03:54.498 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ sleep 0.1 2023-08-26T14:03:54.600 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ test_image_present cluster1 mirror image_4 present 12c07056e5c1 2023-08-26T14:03:54.600 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster1 2023-08-26T14:03:54.600 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T14:03:54.600 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=image_4 2023-08-26T14:03:54.600 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local test_state=present 2023-08-26T14:03:54.601 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image_id=12c07056e5c1 2023-08-26T14:03:54.601 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local current_state=deleted 2023-08-26T14:03:54.601 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local current_image_id 2023-08-26T14:03:54.601 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ get_image_id cluster1 mirror image_4 2023-08-26T14:03:54.601 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local cluster=cluster1 2023-08-26T14:03:54.602 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local pool=mirror 2023-08-26T14:03:54.602 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local image=image_4 2023-08-26T14:03:54.602 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ rbd --cluster=cluster1 info mirror/image_4 2023-08-26T14:03:54.602 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2023-08-26T14:03:54.644 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ current_image_id=12c07056e5c1 2023-08-26T14:03:54.644 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ test -n 12c07056e5c1 2023-08-26T14:03:54.644 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ test -z 12c07056e5c1 -o 12c07056e5c1 = 12c07056e5c1 2023-08-26T14:03:54.644 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ current_state=present 2023-08-26T14:03:54.644 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ test present = present 2023-08-26T14:03:54.645 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ return 0 2023-08-26T14:03:54.645 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ disable_mirror cluster2 mirror image_4 2023-08-26T14:03:54.645 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster2 2023-08-26T14:03:54.645 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T14:03:54.645 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=image_4 2023-08-26T14:03:54.646 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rbd --cluster=cluster2 mirror image disable mirror/image_4 2023-08-26T14:03:55.292 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi124.stderr:2023-08-26T14:03:55.291+0000 7f0338336700 -1 rbd::mirror::image_replayer::snapshot::Replayer: 0x55798c8bfc00 scan_remote_mirror_snapshots: failed to locate remote start snapshot: snap_id=374 2023-08-26T14:03:55.293 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi124.stderr:2023-08-26T14:03:55.291+0000 7f0338336700 -1 rbd::mirror::image_replayer::snapshot::Replayer: 0x55798c8bfc00 scan_remote_mirror_snapshots: split-brain detected: failed to find matching non-primary snapshot in remote image: local_snap_id_start=374, local_snap_ns=[mirror state=non-primary, complete=1, mirror_peer_uuids=, primary_mirror_uuid=d88cafc5-6d65-4864-9230-fe3ef515f83d, primary_snap_id=176, last_copied_object_number=128, snap_seqs={374=18446744073709551614}] 2023-08-26T14:03:58.061 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi124.stderr:2023-08-26T14:03:58.060+0000 7fc24b3d6700 -1 rbd::mirror::ImageMap: 0x555d5b4defc0 handle_peer_ack_remove: failed to remove global_image_id=0df27f7e-c961-4a0f-a3f7-f081874bb9b5 2023-08-26T14:03:59.597 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.0 is failed for ~97s 2023-08-26T14:04:00.295 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:Mirroring disabled 2023-08-26T14:04:00.302 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ wait_for_image_present cluster1 mirror image_4 deleted 2023-08-26T14:04:00.303 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster1 2023-08-26T14:04:00.303 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T14:04:00.303 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=image_4 2023-08-26T14:04:00.303 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local state=deleted 2023-08-26T14:04:00.303 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image_id= 2023-08-26T14:04:00.304 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local s 2023-08-26T14:04:00.304 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ test -n '' 2023-08-26T14:04:00.304 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ get_image_id cluster1 mirror image_4 2023-08-26T14:04:00.304 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local cluster=cluster1 2023-08-26T14:04:00.304 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local pool=mirror 2023-08-26T14:04:00.304 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local image=image_4 2023-08-26T14:04:00.305 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ rbd --cluster=cluster1 info mirror/image_4 2023-08-26T14:04:00.305 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2023-08-26T14:04:00.339 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:rbd: error opening image image_4: (2) No such file or directory 2023-08-26T14:04:00.343 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ image_id= 2023-08-26T14:04:00.343 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2023-08-26T14:04:00.343 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ sleep 0.1 2023-08-26T14:04:00.444 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ test_image_present cluster1 mirror image_4 deleted '' 2023-08-26T14:04:00.445 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster1 2023-08-26T14:04:00.445 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T14:04:00.445 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=image_4 2023-08-26T14:04:00.445 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local test_state=deleted 2023-08-26T14:04:00.446 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image_id= 2023-08-26T14:04:00.446 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local current_state=deleted 2023-08-26T14:04:00.446 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local current_image_id 2023-08-26T14:04:00.446 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ get_image_id cluster1 mirror image_4 2023-08-26T14:04:00.446 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local cluster=cluster1 2023-08-26T14:04:00.447 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local pool=mirror 2023-08-26T14:04:00.447 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local image=image_4 2023-08-26T14:04:00.447 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ rbd --cluster=cluster1 info mirror/image_4 2023-08-26T14:04:00.447 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2023-08-26T14:04:00.479 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:rbd: error opening image image_4: (2) No such file or directory 2023-08-26T14:04:00.482 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ current_image_id= 2023-08-26T14:04:00.482 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ test -n '' 2023-08-26T14:04:00.482 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ test deleted = deleted 2023-08-26T14:04:00.483 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ return 0 2023-08-26T14:04:00.483 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ purge_snapshots cluster2 mirror image_4 2023-08-26T14:04:00.483 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster2 2023-08-26T14:04:00.483 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T14:04:00.483 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=image_4 2023-08-26T14:04:00.484 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rbd --cluster cluster2 snap purge mirror/image_4 2023-08-26T14:04:00.527 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ remove_image_retry cluster2 mirror image_4 2023-08-26T14:04:00.527 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster2 2023-08-26T14:04:00.527 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T14:04:00.528 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=image_4 2023-08-26T14:04:00.528 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ for s in 0 1 2 4 8 16 32 2023-08-26T14:04:00.528 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ sleep 0 2023-08-26T14:04:00.529 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ remove_image cluster2 mirror image_4 2023-08-26T14:04:00.529 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster2 2023-08-26T14:04:00.529 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T14:04:00.529 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=image_4 2023-08-26T14:04:00.529 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_4 2023-08-26T14:04:00.570 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rbd --cluster=cluster2 rm mirror/image_4 2023-08-26T14:04:05.001 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.0 is failed for ~103s 2023-08-26T14:04:10.406 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.0 is failed for ~108s 2023-08-26T14:04:15.810 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.0 is failed for ~113s 2023-08-26T14:04:21.214 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.0 is failed for ~119s 2023-08-26T14:04:24.608 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr: Removing image: 1% complete... Removing image: 2% complete... Removing image: 3% complete... Removing image: 4% complete... Removing image: 5% complete... Removing image: 6% complete... Removing image: 7% complete... Removing image: 8% complete... Removing image: 9% complete... Removing image: 10% complete... Removing image: 11% complete... Removing image: 12% complete... Removing image: 13% complete... Removing image: 14% complete... Removing image: 15% complete... Removing image: 16% complete... Removing image: 17% complete... Removing image: 18% complete... Removing image: 19% complete... Removing image: 20% complete... Removing image: 21% complete... Removing image: 22% complete... Removing image: 23% complete... Removing image: 24% complete... Removing image: 25% complete... Removing image: 26% complete... Removing image: 27% complete... Removing image: 28% complete... Removing image: 29% complete... Removing image: 30% complete... Removing image: 31% complete... Removing image: 32% complete... Removing image: 33% complete... Removing image: 34% complete... Removing image: 35% complete... Removing image: 36% complete... Removing image: 37% complete... Removing image: 38% complete... Removing image: 39% complete... Removing image: 40% complete... Removing image: 41% complete... Removing image: 42% complete... Removing image: 43% complete... Removing image: 44% complete... Removing image: 45% complete... Removing image: 46% complete... Removing image: 47% complete... Removing image: 48% complete... Removing image: 49% complete... Removing image: 50% complete... Removing image: 51% complete... Removing image: 52% complete... Removing image: 53% complete... Removing image: 54% complete... Removing image: 55% complete... Removing image: 56% complete... Removing image: 57% complete... Removing image: 58% complete... Removing image: 59% complete... Removing image: 60% complete... Removing image: 61% complete... Removing image: 62% complete... Removing image: 63% complete... Removing image: 64% complete... Removing image: 65% complete... Removing image: 66% complete... Removing image: 67% complete... Removing image: 68% complete... Removing image: 69% complete... Removing image: 70% complete... Removing image: 71% complete... Removing image: 72% complete... Removing image: 73% complete... Removing image: 74% complete... Removing image: 75% complete... Removing image: 76% complete... Removing image: 77% complete... Removing image: 78% complete... Removing image: 79% complete... Removing image: 80% complete... Removing image: 81% complete... Removing image: 82% complete... Removing image: 83% complete... Removing image: 84% complete... Removing image: 85% complete... Removing image: 86% complete... Removing image: 87% complete... Removing image: 88% complete... Removing image: 89% complete... Removing image: 90% complete... Removing image: 91% complete... Removing image: 92% complete... Removing image: 93% complete... Removing image: 94% complete... Removing image: 95% complete... Removing image: 96% complete... Removing image: 97% complete... Removing image: 98% complete... Removing image: 99% complete... Removing image: 100% complete...done. 2023-08-26T14:04:24.611 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ return 0 2023-08-26T14:04:24.612 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ for i in `seq 1 10` 2023-08-26T14:04:24.612 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ image=image_5 2023-08-26T14:04:24.612 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ create_image cluster2 mirror image_5 512M 2023-08-26T14:04:24.612 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster2 2023-08-26T14:04:24.612 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ shift 2023-08-26T14:04:24.613 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T14:04:24.613 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ shift 2023-08-26T14:04:24.613 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=image_5 2023-08-26T14:04:24.613 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ shift 2023-08-26T14:04:24.613 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local size=128 2023-08-26T14:04:24.613 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ '[' -n 512M ']' 2023-08-26T14:04:24.614 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ size=512M 2023-08-26T14:04:24.614 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ shift 2023-08-26T14:04:24.614 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rbd --cluster cluster2 create --size 512M --image-feature layering,exclusive-lock mirror/image_5 2023-08-26T14:04:24.657 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ enable_mirror cluster2 mirror image_5 2023-08-26T14:04:24.657 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster2 2023-08-26T14:04:24.657 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T14:04:24.657 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=image_5 2023-08-26T14:04:24.658 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local mode=snapshot 2023-08-26T14:04:24.658 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_5 snapshot 2023-08-26T14:04:25.421 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:Mirroring enabled 2023-08-26T14:04:25.429 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rbd --cluster=cluster2 info mirror/image_5 2023-08-26T14:04:25.468 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:rbd image 'image_5': 2023-08-26T14:04:25.468 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: size 512 MiB in 128 objects 2023-08-26T14:04:25.469 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: order 22 (4 MiB objects) 2023-08-26T14:04:25.469 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: snapshot_count: 1 2023-08-26T14:04:25.469 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: id: 18cbcfb6791 2023-08-26T14:04:25.469 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: block_name_prefix: rbd_data.18cbcfb6791 2023-08-26T14:04:25.469 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: format: 2 2023-08-26T14:04:25.470 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: features: layering, exclusive-lock 2023-08-26T14:04:25.470 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: op_features: 2023-08-26T14:04:25.470 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: flags: 2023-08-26T14:04:25.470 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: create_timestamp: Sat Aug 26 14:04:24 2023 2023-08-26T14:04:25.470 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: access_timestamp: Sat Aug 26 14:04:24 2023 2023-08-26T14:04:25.471 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: modify_timestamp: Sat Aug 26 14:04:24 2023 2023-08-26T14:04:25.471 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: mirroring state: enabled 2023-08-26T14:04:25.471 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: mirroring mode: snapshot 2023-08-26T14:04:25.471 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: mirroring global id: b9e1bf66-dac3-4164-a465-3e1a4e5a3598 2023-08-26T14:04:25.471 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: mirroring primary: true 2023-08-26T14:04:25.472 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ stress_write_image cluster2 mirror image_5 2023-08-26T14:04:25.472 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster2 2023-08-26T14:04:25.472 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T14:04:25.473 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=image_5 2023-08-26T14:04:25.473 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2023-08-26T14:04:25.473 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local duration=13 2023-08-26T14:04:25.474 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ set +e 2023-08-26T14:04:25.474 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ timeout 13s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_5 --debug-rbd=20 --debug-journaler=20 2023-08-26T14:04:25.620 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:100 IOs, 26816512 bytes 2023-08-26T14:04:25.716 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:200 IOs, 48560128 bytes 2023-08-26T14:04:25.863 INFO:tasks.ceph.cluster2.osd.2.smithi124.stderr:2023-08-26T14:04:25.861+0000 7fb790854700 -1 reset not still connected to 0x5598e378aea0 2023-08-26T14:04:25.864 INFO:tasks.ceph.cluster2.osd.2.smithi124.stderr:2023-08-26T14:04:25.863+0000 7fb790854700 -1 reset not still connected to 0x5598f06d3a00 2023-08-26T14:04:25.865 INFO:tasks.ceph.cluster2.osd.2.smithi124.stderr:2023-08-26T14:04:25.863+0000 7fb790854700 -1 reset not still connected to 0x5599107b80d0 2023-08-26T14:04:25.865 INFO:tasks.ceph.cluster2.osd.2.smithi124.stderr:2023-08-26T14:04:25.863+0000 7fb790854700 -1 reset not still connected to 0x5599136fe1a0 2023-08-26T14:04:25.865 INFO:tasks.ceph.cluster2.osd.2.smithi124.stderr:2023-08-26T14:04:25.863+0000 7fb790854700 -1 reset not still connected to 0x5599136fe680 2023-08-26T14:04:25.881 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:300 IOs, 93319168 bytes 2023-08-26T14:04:26.003 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:400 IOs, 160437248 bytes 2023-08-26T14:04:26.169 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:500 IOs, 204395520 bytes 2023-08-26T14:04:26.287 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:600 IOs, 244626432 bytes 2023-08-26T14:04:26.430 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:700 IOs, 283297792 bytes 2023-08-26T14:04:26.616 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:800 IOs, 367660032 bytes 2023-08-26T14:04:26.618 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.0 is failed for ~124s 2023-08-26T14:04:26.728 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:900 IOs, 398440448 bytes 2023-08-26T14:04:26.870 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:1000 IOs, 463376384 bytes 2023-08-26T14:04:27.005 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:1100 IOs, 511096832 bytes 2023-08-26T14:04:27.114 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:1200 IOs, 533799936 bytes 2023-08-26T14:04:27.268 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:1300 IOs, 606419968 bytes 2023-08-26T14:04:27.406 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:1400 IOs, 649461760 bytes 2023-08-26T14:04:27.548 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:1500 IOs, 700525568 bytes 2023-08-26T14:04:27.716 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:1600 IOs, 768101376 bytes 2023-08-26T14:04:27.881 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:1700 IOs, 823297024 bytes 2023-08-26T14:04:28.024 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:1800 IOs, 876673024 bytes 2023-08-26T14:04:28.159 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:1900 IOs, 913905664 bytes 2023-08-26T14:04:28.275 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:2000 IOs, 953619456 bytes 2023-08-26T14:04:28.386 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:2100 IOs, 972054528 bytes 2023-08-26T14:04:28.533 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:2200 IOs, 1018058752 bytes 2023-08-26T14:04:28.656 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:2300 IOs, 1059781632 bytes 2023-08-26T14:04:28.728 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:2400 IOs, 1060413440 bytes 2023-08-26T14:04:28.885 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:2500 IOs, 1122695168 bytes 2023-08-26T14:04:28.996 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:2600 IOs, 1147204608 bytes 2023-08-26T14:04:29.141 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:2700 IOs, 1199994880 bytes 2023-08-26T14:04:29.341 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:2800 IOs, 1286057984 bytes 2023-08-26T14:04:29.424 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:2900 IOs, 1298211840 bytes 2023-08-26T14:04:29.581 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:3000 IOs, 1353155584 bytes 2023-08-26T14:04:29.661 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:3100 IOs, 1361992704 bytes 2023-08-26T14:04:29.861 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:3200 IOs, 1443567616 bytes 2023-08-26T14:04:29.958 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:3300 IOs, 1468094464 bytes 2023-08-26T14:04:30.079 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:3400 IOs, 1526172672 bytes 2023-08-26T14:04:30.215 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:3500 IOs, 1556795392 bytes 2023-08-26T14:04:30.323 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:3600 IOs, 1578803200 bytes 2023-08-26T14:04:30.496 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:3700 IOs, 1655486464 bytes 2023-08-26T14:04:30.632 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:3800 IOs, 1699847168 bytes 2023-08-26T14:04:30.759 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:3900 IOs, 1739486208 bytes 2023-08-26T14:04:30.914 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:4000 IOs, 1801266176 bytes 2023-08-26T14:04:31.090 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:4100 IOs, 1882015744 bytes 2023-08-26T14:04:31.187 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:4200 IOs, 1899117568 bytes 2023-08-26T14:04:31.344 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:4300 IOs, 1991098368 bytes 2023-08-26T14:04:31.521 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:4400 IOs, 2040786944 bytes 2023-08-26T14:04:31.661 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:4500 IOs, 2084912128 bytes 2023-08-26T14:04:31.780 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:4600 IOs, 2114290688 bytes 2023-08-26T14:04:31.878 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:4700 IOs, 2141326336 bytes 2023-08-26T14:04:32.006 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:4800 IOs, 2182290432 bytes 2023-08-26T14:04:32.021 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.0 is failed for ~130s 2023-08-26T14:04:32.121 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:4900 IOs, 2219333632 bytes 2023-08-26T14:04:32.218 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:5000 IOs, 2222478336 bytes 2023-08-26T14:04:32.413 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:5100 IOs, 2299254784 bytes 2023-08-26T14:04:32.518 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:5200 IOs, 2318518272 bytes 2023-08-26T14:04:32.613 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:5300 IOs, 2335021056 bytes 2023-08-26T14:04:32.836 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:5400 IOs, 2453402624 bytes 2023-08-26T14:04:33.161 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:5500 IOs, 2505304064 bytes 2023-08-26T14:04:33.252 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:5600 IOs, 2523430912 bytes 2023-08-26T14:04:33.348 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:5700 IOs, 2524020736 bytes 2023-08-26T14:04:33.481 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:5800 IOs, 2575692800 bytes 2023-08-26T14:04:33.575 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:5900 IOs, 2591098880 bytes 2023-08-26T14:04:33.683 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:6000 IOs, 2611616768 bytes 2023-08-26T14:04:33.821 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:6100 IOs, 2657502208 bytes 2023-08-26T14:04:33.943 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:6200 IOs, 2687621120 bytes 2023-08-26T14:04:34.044 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:6300 IOs, 2707154944 bytes 2023-08-26T14:04:34.213 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:6400 IOs, 2785111040 bytes 2023-08-26T14:04:34.388 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:6500 IOs, 2842139648 bytes 2023-08-26T14:04:34.539 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:6600 IOs, 2884441088 bytes 2023-08-26T14:04:34.693 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:6700 IOs, 2941001728 bytes 2023-08-26T14:04:34.843 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:6800 IOs, 2993428480 bytes 2023-08-26T14:04:34.984 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:6900 IOs, 3045081088 bytes 2023-08-26T14:04:35.071 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:7000 IOs, 3065637888 bytes 2023-08-26T14:04:35.219 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:7100 IOs, 3099742208 bytes 2023-08-26T14:04:35.320 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:7200 IOs, 3130506240 bytes 2023-08-26T14:04:35.419 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:7300 IOs, 3143729152 bytes 2023-08-26T14:04:35.527 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:7400 IOs, 3165280256 bytes 2023-08-26T14:04:35.669 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:7500 IOs, 3218745344 bytes 2023-08-26T14:04:35.832 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:7600 IOs, 3278123008 bytes 2023-08-26T14:04:35.980 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:7700 IOs, 3318115328 bytes 2023-08-26T14:04:36.160 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:7800 IOs, 3397544960 bytes 2023-08-26T14:04:36.293 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:7900 IOs, 3443303424 bytes 2023-08-26T14:04:36.396 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:8000 IOs, 3463824384 bytes 2023-08-26T14:04:36.535 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:8100 IOs, 3489734656 bytes 2023-08-26T14:04:36.664 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:8200 IOs, 3516937216 bytes 2023-08-26T14:04:36.770 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:8300 IOs, 3544380416 bytes 2023-08-26T14:04:36.939 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:8400 IOs, 3622146048 bytes 2023-08-26T14:04:37.057 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:8500 IOs, 3660771328 bytes 2023-08-26T14:04:37.273 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:8600 IOs, 3768903680 bytes 2023-08-26T14:04:37.425 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.0 is failed for ~135s 2023-08-26T14:04:37.515 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:8700 IOs, 3852463104 bytes 2023-08-26T14:04:37.601 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:8800 IOs, 3868323840 bytes 2023-08-26T14:04:37.738 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:8900 IOs, 3905607680 bytes 2023-08-26T14:04:37.936 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:9000 IOs, 4003920896 bytes 2023-08-26T14:04:38.062 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:9100 IOs, 4037656576 bytes 2023-08-26T14:04:38.205 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:9200 IOs, 4089544704 bytes 2023-08-26T14:04:38.334 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:9300 IOs, 4121267200 bytes 2023-08-26T14:04:38.443 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:9400 IOs, 4150909952 bytes 2023-08-26T14:04:38.480 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ error_code=124 2023-08-26T14:04:38.480 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ set -e 2023-08-26T14:04:38.481 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ '[' 124 -eq 124 ']' 2023-08-26T14:04:38.481 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ return 0 2023-08-26T14:04:38.481 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ wait_for_image_present cluster1 mirror image_5 present 2023-08-26T14:04:38.481 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster1 2023-08-26T14:04:38.482 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T14:04:38.482 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=image_5 2023-08-26T14:04:38.482 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local state=present 2023-08-26T14:04:38.483 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image_id= 2023-08-26T14:04:38.483 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local s 2023-08-26T14:04:38.483 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ test -n '' 2023-08-26T14:04:38.483 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ get_image_id cluster1 mirror image_5 2023-08-26T14:04:38.484 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local cluster=cluster1 2023-08-26T14:04:38.484 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local pool=mirror 2023-08-26T14:04:38.484 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local image=image_5 2023-08-26T14:04:38.484 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ rbd --cluster=cluster1 info mirror/image_5 2023-08-26T14:04:38.485 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2023-08-26T14:04:38.522 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ image_id=12c0dfc182cf 2023-08-26T14:04:38.522 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2023-08-26T14:04:38.522 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ sleep 0.1 2023-08-26T14:04:38.624 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ test_image_present cluster1 mirror image_5 present 12c0dfc182cf 2023-08-26T14:04:38.624 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster1 2023-08-26T14:04:38.624 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T14:04:38.624 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=image_5 2023-08-26T14:04:38.625 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local test_state=present 2023-08-26T14:04:38.625 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image_id=12c0dfc182cf 2023-08-26T14:04:38.625 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local current_state=deleted 2023-08-26T14:04:38.625 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local current_image_id 2023-08-26T14:04:38.625 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ get_image_id cluster1 mirror image_5 2023-08-26T14:04:38.626 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local cluster=cluster1 2023-08-26T14:04:38.626 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local pool=mirror 2023-08-26T14:04:38.626 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local image=image_5 2023-08-26T14:04:38.626 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ rbd --cluster=cluster1 info mirror/image_5 2023-08-26T14:04:38.626 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2023-08-26T14:04:38.667 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ current_image_id=12c0dfc182cf 2023-08-26T14:04:38.667 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ test -n 12c0dfc182cf 2023-08-26T14:04:38.667 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ test -z 12c0dfc182cf -o 12c0dfc182cf = 12c0dfc182cf 2023-08-26T14:04:38.668 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ current_state=present 2023-08-26T14:04:38.668 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ test present = present 2023-08-26T14:04:38.668 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ return 0 2023-08-26T14:04:38.668 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ disable_mirror cluster2 mirror image_5 2023-08-26T14:04:38.668 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster2 2023-08-26T14:04:38.669 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T14:04:38.669 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=image_5 2023-08-26T14:04:38.669 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rbd --cluster=cluster2 mirror image disable mirror/image_5 2023-08-26T14:04:39.410 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi124.stderr:2023-08-26T14:04:39.408+0000 7f0338b37700 -1 rbd::mirror::image_replayer::snapshot::Replayer: 0x55798ca02c00 scan_remote_mirror_snapshots: failed to locate remote start snapshot: snap_id=376 2023-08-26T14:04:39.410 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi124.stderr:2023-08-26T14:04:39.408+0000 7f0338b37700 -1 rbd::mirror::image_replayer::snapshot::Replayer: 0x55798ca02c00 scan_remote_mirror_snapshots: split-brain detected: failed to find matching non-primary snapshot in remote image: local_snap_id_start=376, local_snap_ns=[mirror state=non-primary, complete=1, mirror_peer_uuids=, primary_mirror_uuid=d88cafc5-6d65-4864-9230-fe3ef515f83d, primary_snap_id=178, last_copied_object_number=128, snap_seqs={376=18446744073709551614}] 2023-08-26T14:04:42.829 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.0 is failed for ~140s 2023-08-26T14:04:44.413 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:Mirroring disabled 2023-08-26T14:04:44.422 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ wait_for_image_present cluster1 mirror image_5 deleted 2023-08-26T14:04:44.422 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster1 2023-08-26T14:04:44.422 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T14:04:44.423 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=image_5 2023-08-26T14:04:44.423 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local state=deleted 2023-08-26T14:04:44.423 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image_id= 2023-08-26T14:04:44.423 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local s 2023-08-26T14:04:44.424 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ test -n '' 2023-08-26T14:04:44.424 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ get_image_id cluster1 mirror image_5 2023-08-26T14:04:44.424 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local cluster=cluster1 2023-08-26T14:04:44.424 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local pool=mirror 2023-08-26T14:04:44.425 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local image=image_5 2023-08-26T14:04:44.425 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ rbd --cluster=cluster1 info mirror/image_5 2023-08-26T14:04:44.425 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2023-08-26T14:04:44.461 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:rbd: error opening image image_5: (2) No such file or directory 2023-08-26T14:04:44.465 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ image_id= 2023-08-26T14:04:44.465 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2023-08-26T14:04:44.465 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ sleep 0.1 2023-08-26T14:04:44.567 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ test_image_present cluster1 mirror image_5 deleted '' 2023-08-26T14:04:44.567 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster1 2023-08-26T14:04:44.567 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T14:04:44.567 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=image_5 2023-08-26T14:04:44.568 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local test_state=deleted 2023-08-26T14:04:44.568 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image_id= 2023-08-26T14:04:44.568 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local current_state=deleted 2023-08-26T14:04:44.568 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local current_image_id 2023-08-26T14:04:44.568 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ get_image_id cluster1 mirror image_5 2023-08-26T14:04:44.569 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local cluster=cluster1 2023-08-26T14:04:44.569 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local pool=mirror 2023-08-26T14:04:44.569 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local image=image_5 2023-08-26T14:04:44.569 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ rbd --cluster=cluster1 info mirror/image_5 2023-08-26T14:04:44.569 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2023-08-26T14:04:44.602 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:rbd: error opening image image_5: (2) No such file or directory 2023-08-26T14:04:44.605 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ current_image_id= 2023-08-26T14:04:44.606 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ test -n '' 2023-08-26T14:04:44.606 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ test deleted = deleted 2023-08-26T14:04:44.606 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ return 0 2023-08-26T14:04:44.606 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ purge_snapshots cluster2 mirror image_5 2023-08-26T14:04:44.607 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster2 2023-08-26T14:04:44.607 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T14:04:44.607 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=image_5 2023-08-26T14:04:44.607 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rbd --cluster cluster2 snap purge mirror/image_5 2023-08-26T14:04:44.651 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ remove_image_retry cluster2 mirror image_5 2023-08-26T14:04:44.651 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster2 2023-08-26T14:04:44.651 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T14:04:44.652 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=image_5 2023-08-26T14:04:44.652 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ for s in 0 1 2 4 8 16 32 2023-08-26T14:04:44.652 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ sleep 0 2023-08-26T14:04:44.653 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ remove_image cluster2 mirror image_5 2023-08-26T14:04:44.653 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster2 2023-08-26T14:04:44.653 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T14:04:44.653 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=image_5 2023-08-26T14:04:44.654 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_5 2023-08-26T14:04:44.696 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rbd --cluster=cluster2 rm mirror/image_5 2023-08-26T14:04:48.233 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.0 is failed for ~146s 2023-08-26T14:04:49.777 INFO:tasks.rbd_mirror_thrash:kill cluster1.client.mirror.0 2023-08-26T14:04:49.777 INFO:tasks.rbd_mirror.cluster1.client.mirror.0:Sent signal 15 2023-08-26T14:04:49.778 INFO:tasks.rbd_mirror_thrash:waiting for 9 secs before reviving daemons 2023-08-26T14:04:49.779 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi124.stderr:2023-08-26T14:04:49.778+0000 7f466606e700 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.0 (PID: 133324) UID: 1000 2023-08-26T14:04:53.638 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.0 is failed for ~151s 2023-08-26T14:04:54.561 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.0 with skip (0.7161039914324936) > weight (0.25) 2023-08-26T14:04:54.561 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.1 with skip (0.257891702969429) > weight (0.25) 2023-08-26T14:04:54.561 INFO:tasks.rbd_mirror_thrash:kill cluster2.client.mirror.2 2023-08-26T14:04:54.562 INFO:tasks.rbd_mirror.cluster2.client.mirror.2:Sent signal 15 2023-08-26T14:04:54.562 INFO:tasks.rbd_mirror_thrash:waiting for 9 secs before reviving daemons 2023-08-26T14:04:54.563 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi124.stderr:2023-08-26T14:04:54.562+0000 7fcd4a5be700 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.2 (PID: 119870) UID: 1000 2023-08-26T14:04:58.779 INFO:tasks.rbd_mirror_thrash:waiting for cluster1.client.mirror.0 2023-08-26T14:04:58.779 INFO:teuthology.orchestra.run:waiting for 600 2023-08-26T14:04:58.779 INFO:tasks.rbd_mirror_thrash:reviving cluster1.client.mirror.0 2023-08-26T14:04:58.780 INFO:tasks.rbd_mirror.cluster1.client.mirror.0:Restarting daemon 2023-08-26T14:04:58.780 DEBUG:teuthology.orchestra.run.smithi124:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.0 2023-08-26T14:04:58.783 INFO:tasks.rbd_mirror.cluster1.client.mirror.0:Started 2023-08-26T14:04:58.783 INFO:tasks.rbd_mirror_thrash:waiting for 119 secs before thrashing 2023-08-26T14:04:59.041 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.0 is failed for ~157s 2023-08-26T14:05:03.562 INFO:tasks.rbd_mirror_thrash:waiting for cluster2.client.mirror.2 2023-08-26T14:05:03.563 INFO:teuthology.orchestra.run:waiting for 600 2023-08-26T14:05:03.563 INFO:tasks.rbd_mirror_thrash:reviving cluster2.client.mirror.2 2023-08-26T14:05:03.563 INFO:tasks.rbd_mirror.cluster2.client.mirror.2:Restarting daemon 2023-08-26T14:05:03.564 DEBUG:teuthology.orchestra.run.smithi124:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.2 2023-08-26T14:05:03.566 INFO:tasks.rbd_mirror.cluster2.client.mirror.2:Started 2023-08-26T14:05:03.566 INFO:tasks.rbd_mirror_thrash:waiting for 119 secs before thrashing 2023-08-26T14:05:04.445 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.0 is failed for ~162s 2023-08-26T14:05:06.650 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr: Removing image: 1% complete... Removing image: 2% complete... Removing image: 3% complete... Removing image: 4% complete... Removing image: 5% complete... Removing image: 6% complete... Removing image: 7% complete... Removing image: 8% complete... Removing image: 9% complete... Removing image: 10% complete... Removing image: 11% complete... Removing image: 12% complete... Removing image: 13% complete... Removing image: 14% complete... Removing image: 15% complete... Removing image: 16% complete... Removing image: 17% complete... Removing image: 18% complete... Removing image: 19% complete... Removing image: 20% complete... Removing image: 21% complete... Removing image: 22% complete... Removing image: 23% complete... Removing image: 24% complete... Removing image: 25% complete... Removing image: 26% complete... Removing image: 27% complete... Removing image: 28% complete... Removing image: 29% complete... Removing image: 30% complete... Removing image: 31% complete... Removing image: 32% complete... Removing image: 33% complete... Removing image: 34% complete... Removing image: 35% complete... Removing image: 36% complete... Removing image: 37% complete... Removing image: 38% complete... Removing image: 39% complete... Removing image: 40% complete... Removing image: 41% complete... Removing image: 42% complete... Removing image: 43% complete... Removing image: 44% complete... Removing image: 45% complete... Removing image: 46% complete... Removing image: 47% complete... Removing image: 48% complete... Removing image: 49% complete... Removing image: 50% complete... Removing image: 51% complete... Removing image: 52% complete... Removing image: 53% complete... Removing image: 54% complete... Removing image: 55% complete... Removing image: 56% complete... Removing image: 57% complete... Removing image: 58% complete... Removing image: 59% complete... Removing image: 60% complete... Removing image: 61% complete... Removing image: 62% complete... Removing image: 63% complete... Removing image: 64% complete... Removing image: 65% complete... Removing image: 66% complete... Removing image: 67% complete... Removing image: 68% complete... Removing image: 69% complete... Removing image: 70% complete... Removing image: 71% complete... Removing image: 72% complete... Removing image: 73% complete... Removing image: 74% complete... Removing image: 75% complete... Removing image: 76% complete... Removing image: 77% complete... Removing image: 78% complete... Removing image: 79% complete... Removing image: 80% complete... Removing image: 81% complete... Removing image: 82% complete... Removing image: 83% complete... Removing image: 84% complete... Removing image: 85% complete... Removing image: 86% complete... Removing image: 87% complete... Removing image: 88% complete... Removing image: 89% complete... Removing image: 90% complete... Removing image: 91% complete... Removing image: 92% complete... Removing image: 93% complete... Removing image: 94% complete... Removing image: 95% complete... Removing image: 96% complete... Removing image: 97% complete... Removing image: 98% complete... Removing image: 99% complete... Removing image: 100% complete...done. 2023-08-26T14:05:06.654 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ return 0 2023-08-26T14:05:06.654 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ for i in `seq 1 10` 2023-08-26T14:05:06.655 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ image=image_6 2023-08-26T14:05:06.655 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ create_image cluster2 mirror image_6 512M 2023-08-26T14:05:06.655 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster2 2023-08-26T14:05:06.655 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ shift 2023-08-26T14:05:06.656 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T14:05:06.656 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ shift 2023-08-26T14:05:06.656 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=image_6 2023-08-26T14:05:06.656 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ shift 2023-08-26T14:05:06.656 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local size=128 2023-08-26T14:05:06.657 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ '[' -n 512M ']' 2023-08-26T14:05:06.657 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ size=512M 2023-08-26T14:05:06.657 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ shift 2023-08-26T14:05:06.657 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rbd --cluster cluster2 create --size 512M --image-feature layering,exclusive-lock mirror/image_6 2023-08-26T14:05:06.699 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ enable_mirror cluster2 mirror image_6 2023-08-26T14:05:06.700 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster2 2023-08-26T14:05:06.700 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T14:05:06.700 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=image_6 2023-08-26T14:05:06.700 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local mode=snapshot 2023-08-26T14:05:06.701 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_6 snapshot 2023-08-26T14:05:07.483 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:Mirroring enabled 2023-08-26T14:05:07.490 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rbd --cluster=cluster2 info mirror/image_6 2023-08-26T14:05:07.527 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:rbd image 'image_6': 2023-08-26T14:05:07.528 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: size 512 MiB in 128 objects 2023-08-26T14:05:07.528 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: order 22 (4 MiB objects) 2023-08-26T14:05:07.528 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: snapshot_count: 1 2023-08-26T14:05:07.528 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: id: 18f17130ad0d 2023-08-26T14:05:07.529 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: block_name_prefix: rbd_data.18f17130ad0d 2023-08-26T14:05:07.529 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: format: 2 2023-08-26T14:05:07.529 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: features: layering, exclusive-lock 2023-08-26T14:05:07.529 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: op_features: 2023-08-26T14:05:07.529 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: flags: 2023-08-26T14:05:07.529 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: create_timestamp: Sat Aug 26 14:05:06 2023 2023-08-26T14:05:07.530 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: access_timestamp: Sat Aug 26 14:05:06 2023 2023-08-26T14:05:07.530 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: modify_timestamp: Sat Aug 26 14:05:06 2023 2023-08-26T14:05:07.530 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: mirroring state: enabled 2023-08-26T14:05:07.530 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: mirroring mode: snapshot 2023-08-26T14:05:07.531 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: mirroring global id: 6605bd4c-ac94-49ce-b4f9-5cb2f8e9f0ad 2023-08-26T14:05:07.531 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: mirroring primary: true 2023-08-26T14:05:07.532 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ stress_write_image cluster2 mirror image_6 2023-08-26T14:05:07.532 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster2 2023-08-26T14:05:07.532 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T14:05:07.532 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=image_6 2023-08-26T14:05:07.533 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2023-08-26T14:05:07.533 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local duration=11 2023-08-26T14:05:07.533 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ set +e 2023-08-26T14:05:07.533 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ timeout 11s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_6 --debug-rbd=20 --debug-journaler=20 2023-08-26T14:05:07.736 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:100 IOs, 62246912 bytes 2023-08-26T14:05:07.833 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:200 IOs, 76445696 bytes 2023-08-26T14:05:08.002 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:300 IOs, 148020224 bytes 2023-08-26T14:05:08.085 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:400 IOs, 148637696 bytes 2023-08-26T14:05:08.184 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:500 IOs, 161321984 bytes 2023-08-26T14:05:08.307 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:600 IOs, 195634176 bytes 2023-08-26T14:05:08.459 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:700 IOs, 250476544 bytes 2023-08-26T14:05:08.608 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:800 IOs, 315827200 bytes 2023-08-26T14:05:08.781 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:900 IOs, 375077888 bytes 2023-08-26T14:05:08.894 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:1000 IOs, 400188416 bytes 2023-08-26T14:05:09.037 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:1100 IOs, 443618304 bytes 2023-08-26T14:05:09.220 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:1200 IOs, 512923648 bytes 2023-08-26T14:05:09.339 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:1300 IOs, 553931776 bytes 2023-08-26T14:05:09.440 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:1400 IOs, 573869056 bytes 2023-08-26T14:05:09.546 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:1500 IOs, 624973824 bytes 2023-08-26T14:05:09.655 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:1600 IOs, 642401280 bytes 2023-08-26T14:05:09.769 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:1700 IOs, 667697152 bytes 2023-08-26T14:05:09.848 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.0 is failed for ~168s 2023-08-26T14:05:09.899 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:1800 IOs, 699500544 bytes 2023-08-26T14:05:10.073 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:1900 IOs, 769224704 bytes 2023-08-26T14:05:10.233 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:2000 IOs, 835605504 bytes 2023-08-26T14:05:10.330 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:2100 IOs, 852228096 bytes 2023-08-26T14:05:10.464 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:2200 IOs, 889816064 bytes 2023-08-26T14:05:10.607 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:2300 IOs, 944194560 bytes 2023-08-26T14:05:10.700 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:2400 IOs, 958058496 bytes 2023-08-26T14:05:10.816 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:2500 IOs, 967335936 bytes 2023-08-26T14:05:10.962 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:2600 IOs, 1020070912 bytes 2023-08-26T14:05:11.108 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:2700 IOs, 1069788160 bytes 2023-08-26T14:05:11.268 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:2800 IOs, 1110885376 bytes 2023-08-26T14:05:11.388 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:2900 IOs, 1152624640 bytes 2023-08-26T14:05:11.510 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:3000 IOs, 1179019264 bytes 2023-08-26T14:05:11.663 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:3100 IOs, 1252763648 bytes 2023-08-26T14:05:11.784 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:3200 IOs, 1270650880 bytes 2023-08-26T14:05:11.917 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:3300 IOs, 1314882560 bytes 2023-08-26T14:05:12.111 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:3400 IOs, 1394162688 bytes 2023-08-26T14:05:12.250 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:3500 IOs, 1438302208 bytes 2023-08-26T14:05:12.337 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:3600 IOs, 1441471488 bytes 2023-08-26T14:05:12.445 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:3700 IOs, 1472456704 bytes 2023-08-26T14:05:12.539 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:3800 IOs, 1496176640 bytes 2023-08-26T14:05:12.650 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:3900 IOs, 1507465216 bytes 2023-08-26T14:05:12.776 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:4000 IOs, 1540567040 bytes 2023-08-26T14:05:12.886 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:4100 IOs, 1590693888 bytes 2023-08-26T14:05:13.033 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:4200 IOs, 1614917632 bytes 2023-08-26T14:05:13.149 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:4300 IOs, 1641709568 bytes 2023-08-26T14:05:13.274 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:4400 IOs, 1673398272 bytes 2023-08-26T14:05:13.383 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:4500 IOs, 1711023104 bytes 2023-08-26T14:05:13.535 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:4600 IOs, 1755432960 bytes 2023-08-26T14:05:13.660 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:4700 IOs, 1787168768 bytes 2023-08-26T14:05:13.877 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:4800 IOs, 1889083392 bytes 2023-08-26T14:05:14.074 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:4900 IOs, 1987973120 bytes 2023-08-26T14:05:14.257 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:5000 IOs, 2041032704 bytes 2023-08-26T14:05:14.442 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:5100 IOs, 2119963648 bytes 2023-08-26T14:05:14.565 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:5200 IOs, 2157643776 bytes 2023-08-26T14:05:14.663 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:5300 IOs, 2184508416 bytes 2023-08-26T14:05:14.826 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:5400 IOs, 2247853056 bytes 2023-08-26T14:05:14.944 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:5500 IOs, 2286771200 bytes 2023-08-26T14:05:15.086 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:5600 IOs, 2339118080 bytes 2023-08-26T14:05:15.187 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:5700 IOs, 2351515648 bytes 2023-08-26T14:05:15.251 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.0 is failed for ~173s 2023-08-26T14:05:15.368 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:5800 IOs, 2441119744 bytes 2023-08-26T14:05:15.486 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:5900 IOs, 2458271744 bytes 2023-08-26T14:05:15.572 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:6000 IOs, 2461072384 bytes 2023-08-26T14:05:15.726 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:6100 IOs, 2527043584 bytes 2023-08-26T14:05:15.826 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:6200 IOs, 2527674368 bytes 2023-08-26T14:05:15.948 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:6300 IOs, 2567567360 bytes 2023-08-26T14:05:16.068 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:6400 IOs, 2577921024 bytes 2023-08-26T14:05:16.181 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:6500 IOs, 2605229056 bytes 2023-08-26T14:05:16.309 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:6600 IOs, 2646752256 bytes 2023-08-26T14:05:16.470 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:6700 IOs, 2703727616 bytes 2023-08-26T14:05:16.627 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:6800 IOs, 2758935552 bytes 2023-08-26T14:05:16.759 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:6900 IOs, 2798853120 bytes 2023-08-26T14:05:16.923 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:7000 IOs, 2862094336 bytes 2023-08-26T14:05:17.047 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:7100 IOs, 2894566400 bytes 2023-08-26T14:05:17.169 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:7200 IOs, 2914922496 bytes 2023-08-26T14:05:17.292 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:7300 IOs, 2941454336 bytes 2023-08-26T14:05:17.436 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:7400 IOs, 2993991680 bytes 2023-08-26T14:05:17.577 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:7500 IOs, 3043225600 bytes 2023-08-26T14:05:17.720 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:7600 IOs, 3095009280 bytes 2023-08-26T14:05:17.850 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:7700 IOs, 3136192512 bytes 2023-08-26T14:05:17.935 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:7800 IOs, 3136778240 bytes 2023-08-26T14:05:18.052 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:7900 IOs, 3168332800 bytes 2023-08-26T14:05:18.138 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:8000 IOs, 3168925696 bytes 2023-08-26T14:05:18.284 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:8100 IOs, 3225595904 bytes 2023-08-26T14:05:18.414 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:8200 IOs, 3277678592 bytes 2023-08-26T14:05:18.538 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ error_code=124 2023-08-26T14:05:18.538 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ set -e 2023-08-26T14:05:18.538 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ '[' 124 -eq 124 ']' 2023-08-26T14:05:18.539 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ return 0 2023-08-26T14:05:18.539 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ wait_for_image_present cluster1 mirror image_6 present 2023-08-26T14:05:18.539 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster1 2023-08-26T14:05:18.539 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T14:05:18.540 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=image_6 2023-08-26T14:05:18.540 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local state=present 2023-08-26T14:05:18.540 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image_id= 2023-08-26T14:05:18.540 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local s 2023-08-26T14:05:18.540 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ test -n '' 2023-08-26T14:05:18.541 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ get_image_id cluster1 mirror image_6 2023-08-26T14:05:18.541 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local cluster=cluster1 2023-08-26T14:05:18.541 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local pool=mirror 2023-08-26T14:05:18.542 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local image=image_6 2023-08-26T14:05:18.542 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ rbd --cluster=cluster1 info mirror/image_6 2023-08-26T14:05:18.542 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2023-08-26T14:05:18.584 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ image_id=12c0ac6fe0b6 2023-08-26T14:05:18.584 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2023-08-26T14:05:18.584 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ sleep 0.1 2023-08-26T14:05:18.685 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ test_image_present cluster1 mirror image_6 present 12c0ac6fe0b6 2023-08-26T14:05:18.686 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster1 2023-08-26T14:05:18.686 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T14:05:18.686 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=image_6 2023-08-26T14:05:18.686 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local test_state=present 2023-08-26T14:05:18.687 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image_id=12c0ac6fe0b6 2023-08-26T14:05:18.687 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local current_state=deleted 2023-08-26T14:05:18.687 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local current_image_id 2023-08-26T14:05:18.687 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ get_image_id cluster1 mirror image_6 2023-08-26T14:05:18.687 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local cluster=cluster1 2023-08-26T14:05:18.688 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local pool=mirror 2023-08-26T14:05:18.688 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local image=image_6 2023-08-26T14:05:18.688 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ rbd --cluster=cluster1 info mirror/image_6 2023-08-26T14:05:18.688 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2023-08-26T14:05:18.730 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ current_image_id=12c0ac6fe0b6 2023-08-26T14:05:18.731 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ test -n 12c0ac6fe0b6 2023-08-26T14:05:18.731 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ test -z 12c0ac6fe0b6 -o 12c0ac6fe0b6 = 12c0ac6fe0b6 2023-08-26T14:05:18.731 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ current_state=present 2023-08-26T14:05:18.731 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ test present = present 2023-08-26T14:05:18.731 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ return 0 2023-08-26T14:05:18.732 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ disable_mirror cluster2 mirror image_6 2023-08-26T14:05:18.732 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster2 2023-08-26T14:05:18.732 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T14:05:18.732 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=image_6 2023-08-26T14:05:18.732 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rbd --cluster=cluster2 mirror image disable mirror/image_6 2023-08-26T14:05:19.510 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi124.stderr:2023-08-26T14:05:19.508+0000 7f0338336700 -1 rbd::mirror::image_replayer::snapshot::Replayer: 0x55798bacfc00 scan_remote_mirror_snapshots: failed to locate remote start snapshot: snap_id=378 2023-08-26T14:05:19.511 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi124.stderr:2023-08-26T14:05:19.508+0000 7f0338336700 -1 rbd::mirror::image_replayer::snapshot::Replayer: 0x55798bacfc00 scan_remote_mirror_snapshots: split-brain detected: failed to find matching non-primary snapshot in remote image: local_snap_id_start=378, local_snap_ns=[mirror state=non-primary, complete=1, mirror_peer_uuids=, primary_mirror_uuid=d88cafc5-6d65-4864-9230-fe3ef515f83d, primary_snap_id=17a, last_copied_object_number=128, snap_seqs={378=18446744073709551614}] 2023-08-26T14:05:20.655 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.0 is failed for ~178s 2023-08-26T14:05:24.508 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:Mirroring disabled 2023-08-26T14:05:24.517 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ wait_for_image_present cluster1 mirror image_6 deleted 2023-08-26T14:05:24.517 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster1 2023-08-26T14:05:24.518 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T14:05:24.518 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=image_6 2023-08-26T14:05:24.519 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local state=deleted 2023-08-26T14:05:24.519 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image_id= 2023-08-26T14:05:24.519 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local s 2023-08-26T14:05:24.520 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ test -n '' 2023-08-26T14:05:24.520 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ get_image_id cluster1 mirror image_6 2023-08-26T14:05:24.520 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local cluster=cluster1 2023-08-26T14:05:24.521 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local pool=mirror 2023-08-26T14:05:24.521 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local image=image_6 2023-08-26T14:05:24.521 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ rbd --cluster=cluster1 info mirror/image_6 2023-08-26T14:05:24.522 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2023-08-26T14:05:24.555 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:rbd: error opening image image_6: (2) No such file or directory 2023-08-26T14:05:24.558 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ image_id= 2023-08-26T14:05:24.558 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2023-08-26T14:05:24.558 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ sleep 0.1 2023-08-26T14:05:24.660 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ test_image_present cluster1 mirror image_6 deleted '' 2023-08-26T14:05:24.660 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster1 2023-08-26T14:05:24.661 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T14:05:24.661 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=image_6 2023-08-26T14:05:24.661 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local test_state=deleted 2023-08-26T14:05:24.662 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image_id= 2023-08-26T14:05:24.662 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local current_state=deleted 2023-08-26T14:05:24.663 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local current_image_id 2023-08-26T14:05:24.663 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ get_image_id cluster1 mirror image_6 2023-08-26T14:05:24.663 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local cluster=cluster1 2023-08-26T14:05:24.663 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local pool=mirror 2023-08-26T14:05:24.664 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local image=image_6 2023-08-26T14:05:24.664 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ rbd --cluster=cluster1 info mirror/image_6 2023-08-26T14:05:24.664 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2023-08-26T14:05:24.694 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:rbd: error opening image image_6: (2) No such file or directory 2023-08-26T14:05:24.698 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ current_image_id= 2023-08-26T14:05:24.698 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ test -n '' 2023-08-26T14:05:24.698 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ test deleted = deleted 2023-08-26T14:05:24.699 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ return 0 2023-08-26T14:05:24.699 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ purge_snapshots cluster2 mirror image_6 2023-08-26T14:05:24.699 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster2 2023-08-26T14:05:24.700 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T14:05:24.700 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=image_6 2023-08-26T14:05:24.700 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rbd --cluster cluster2 snap purge mirror/image_6 2023-08-26T14:05:24.742 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ remove_image_retry cluster2 mirror image_6 2023-08-26T14:05:24.742 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster2 2023-08-26T14:05:24.742 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T14:05:24.742 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=image_6 2023-08-26T14:05:24.743 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ for s in 0 1 2 4 8 16 32 2023-08-26T14:05:24.743 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ sleep 0 2023-08-26T14:05:24.744 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ remove_image cluster2 mirror image_6 2023-08-26T14:05:24.744 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster2 2023-08-26T14:05:24.744 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T14:05:24.744 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=image_6 2023-08-26T14:05:24.744 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_6 2023-08-26T14:05:24.787 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rbd --cluster=cluster2 rm mirror/image_6 2023-08-26T14:05:26.059 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.0 is failed for ~184s 2023-08-26T14:05:31.463 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.0 is failed for ~189s 2023-08-26T14:05:36.866 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.0 is failed for ~195s 2023-08-26T14:05:42.270 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.0 is failed for ~200s 2023-08-26T14:05:47.674 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.0 is failed for ~205s 2023-08-26T14:05:48.617 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr: Removing image: 1% complete... Removing image: 2% complete... Removing image: 3% complete... Removing image: 4% complete... Removing image: 5% complete... Removing image: 6% complete... Removing image: 7% complete... Removing image: 8% complete... Removing image: 9% complete... Removing image: 10% complete... Removing image: 11% complete... Removing image: 12% complete... Removing image: 13% complete... Removing image: 14% complete... Removing image: 15% complete... Removing image: 16% complete... Removing image: 17% complete... Removing image: 18% complete... Removing image: 19% complete... Removing image: 20% complete... Removing image: 21% complete... Removing image: 22% complete... Removing image: 23% complete... Removing image: 24% complete... Removing image: 25% complete... Removing image: 26% complete... Removing image: 27% complete... Removing image: 28% complete... Removing image: 29% complete... Removing image: 30% complete... Removing image: 31% complete... Removing image: 32% complete... Removing image: 33% complete... Removing image: 34% complete... Removing image: 35% complete... Removing image: 36% complete... Removing image: 37% complete... Removing image: 38% complete... Removing image: 39% complete... Removing image: 40% complete... Removing image: 41% complete... Removing image: 42% complete... Removing image: 43% complete... Removing image: 44% complete... Removing image: 45% complete... Removing image: 46% complete... Removing image: 47% complete... Removing image: 48% complete... Removing image: 49% complete... Removing image: 50% complete... Removing image: 51% complete... Removing image: 52% complete... Removing image: 53% complete... Removing image: 54% complete... Removing image: 55% complete... Removing image: 56% complete... Removing image: 57% complete... Removing image: 58% complete... Removing image: 59% complete... Removing image: 60% complete... Removing image: 61% complete... Removing image: 62% complete... Removing image: 63% complete... Removing image: 64% complete... Removing image: 65% complete... Removing image: 66% complete... Removing image: 67% complete... Removing image: 68% complete... Removing image: 69% complete... Removing image: 70% complete... Removing image: 71% complete... Removing image: 72% complete... Removing image: 73% complete... Removing image: 74% complete... Removing image: 75% complete... Removing image: 76% complete... Removing image: 77% complete... Removing image: 78% complete... Removing image: 79% complete... Removing image: 80% complete... Removing image: 81% complete... Removing image: 82% complete... Removing image: 83% complete... Removing image: 84% complete... Removing image: 85% complete... Removing image: 86% complete... Removing image: 87% complete... Removing image: 88% complete... Removing image: 89% complete... Removing image: 90% complete... Removing image: 91% complete... Removing image: 92% complete... Removing image: 93% complete... Removing image: 94% complete... Removing image: 95% complete... Removing image: 96% complete... Removing image: 97% complete... Removing image: 98% complete... Removing image: 99% complete... Removing image: 100% complete...done. 2023-08-26T14:05:48.621 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ return 0 2023-08-26T14:05:48.621 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ for i in `seq 1 10` 2023-08-26T14:05:48.621 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ image=image_7 2023-08-26T14:05:48.621 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ create_image cluster2 mirror image_7 512M 2023-08-26T14:05:48.622 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster2 2023-08-26T14:05:48.622 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ shift 2023-08-26T14:05:48.622 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T14:05:48.622 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ shift 2023-08-26T14:05:48.622 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=image_7 2023-08-26T14:05:48.623 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ shift 2023-08-26T14:05:48.623 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local size=128 2023-08-26T14:05:48.623 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ '[' -n 512M ']' 2023-08-26T14:05:48.623 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ size=512M 2023-08-26T14:05:48.623 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ shift 2023-08-26T14:05:48.623 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rbd --cluster cluster2 create --size 512M --image-feature layering,exclusive-lock mirror/image_7 2023-08-26T14:05:48.667 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ enable_mirror cluster2 mirror image_7 2023-08-26T14:05:48.668 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster2 2023-08-26T14:05:48.668 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T14:05:48.668 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=image_7 2023-08-26T14:05:48.668 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local mode=snapshot 2023-08-26T14:05:48.668 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_7 snapshot 2023-08-26T14:05:49.451 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:Mirroring enabled 2023-08-26T14:05:49.460 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rbd --cluster=cluster2 info mirror/image_7 2023-08-26T14:05:49.496 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:rbd image 'image_7': 2023-08-26T14:05:49.496 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: size 512 MiB in 128 objects 2023-08-26T14:05:49.496 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: order 22 (4 MiB objects) 2023-08-26T14:05:49.496 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: snapshot_count: 1 2023-08-26T14:05:49.497 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: id: 190952232ea0 2023-08-26T14:05:49.497 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: block_name_prefix: rbd_data.190952232ea0 2023-08-26T14:05:49.497 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: format: 2 2023-08-26T14:05:49.497 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: features: layering, exclusive-lock 2023-08-26T14:05:49.497 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: op_features: 2023-08-26T14:05:49.498 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: flags: 2023-08-26T14:05:49.498 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: create_timestamp: Sat Aug 26 14:05:48 2023 2023-08-26T14:05:49.498 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: access_timestamp: Sat Aug 26 14:05:48 2023 2023-08-26T14:05:49.498 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: modify_timestamp: Sat Aug 26 14:05:48 2023 2023-08-26T14:05:49.498 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: mirroring state: enabled 2023-08-26T14:05:49.499 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: mirroring mode: snapshot 2023-08-26T14:05:49.499 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: mirroring global id: 3ad857df-1a21-4a01-9980-6220b061adb5 2023-08-26T14:05:49.499 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: mirroring primary: true 2023-08-26T14:05:49.500 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ stress_write_image cluster2 mirror image_7 2023-08-26T14:05:49.500 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster2 2023-08-26T14:05:49.500 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T14:05:49.500 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=image_7 2023-08-26T14:05:49.501 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2023-08-26T14:05:49.501 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local duration=9 2023-08-26T14:05:49.501 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ set +e 2023-08-26T14:05:49.502 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ timeout 9s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_7 --debug-rbd=20 --debug-journaler=20 2023-08-26T14:05:49.574 INFO:tasks.ceph.cluster2.osd.1.smithi124.stderr:2023-08-26T14:05:49.573+0000 7f2a62456700 -1 bdev(0x56265f2e0000 /var/lib/ceph/osd/cluster2-1/block) _aio_thread got r=-28 ((28) No space left on device) 2023-08-26T14:05:49.576 INFO:tasks.ceph.cluster2.osd.1.smithi124.stderr:/home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/18.0.0-5706-ge7eb9853/rpm/el8/BUILD/ceph-18.0.0-5706-ge7eb9853/src/blk/kernel/KernelDevice.cc: In function 'void KernelDevice::_aio_thread()' thread 7f2a62456700 time 2023-08-26T14:05:49.573990+0000 2023-08-26T14:05:49.576 INFO:tasks.ceph.cluster2.osd.1.smithi124.stderr:/home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/18.0.0-5706-ge7eb9853/rpm/el8/BUILD/ceph-18.0.0-5706-ge7eb9853/src/blk/kernel/KernelDevice.cc: 633: ceph_abort_msg("Unexpected IO error. This may suggest a hardware issue. Please check your kernel log!") 2023-08-26T14:05:49.576 INFO:tasks.ceph.cluster2.osd.1.smithi124.stderr: ceph version 18.0.0-5706-ge7eb9853 (e7eb9853fddaf661c68f207346131a7e00d8ed2f) reef (dev) 2023-08-26T14:05:49.576 INFO:tasks.ceph.cluster2.osd.1.smithi124.stderr: 1: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string, std::allocator > const&)+0xd7) [0x56265adc49b5] 2023-08-26T14:05:49.576 INFO:tasks.ceph.cluster2.osd.1.smithi124.stderr: 2: (KernelDevice::_aio_thread()+0x1686) [0x56265b9ee416] 2023-08-26T14:05:49.577 INFO:tasks.ceph.cluster2.osd.1.smithi124.stderr: 3: (KernelDevice::AioCompletionThread::entry()+0x11) [0x56265b9f3ed1] 2023-08-26T14:05:49.577 INFO:tasks.ceph.cluster2.osd.1.smithi124.stderr: 4: /lib64/libpthread.so.0(+0x81cf) [0x7f2a74f7d1cf] 2023-08-26T14:05:49.577 INFO:tasks.ceph.cluster2.osd.1.smithi124.stderr: 5: clone() 2023-08-26T14:05:49.577 INFO:tasks.ceph.cluster2.osd.1.smithi124.stderr:2023-08-26T14:05:49.574+0000 7f2a62456700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/18.0.0-5706-ge7eb9853/rpm/el8/BUILD/ceph-18.0.0-5706-ge7eb9853/src/blk/kernel/KernelDevice.cc: In function 'void KernelDevice::_aio_thread()' thread 7f2a62456700 time 2023-08-26T14:05:49.573990+0000 2023-08-26T14:05:49.577 INFO:tasks.ceph.cluster2.osd.1.smithi124.stderr:/home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/18.0.0-5706-ge7eb9853/rpm/el8/BUILD/ceph-18.0.0-5706-ge7eb9853/src/blk/kernel/KernelDevice.cc: 633: ceph_abort_msg("Unexpected IO error. This may suggest a hardware issue. Please check your kernel log!") 2023-08-26T14:05:49.578 INFO:tasks.ceph.cluster2.osd.1.smithi124.stderr: 2023-08-26T14:05:49.578 INFO:tasks.ceph.cluster2.osd.1.smithi124.stderr: ceph version 18.0.0-5706-ge7eb9853 (e7eb9853fddaf661c68f207346131a7e00d8ed2f) reef (dev) 2023-08-26T14:05:49.578 INFO:tasks.ceph.cluster2.osd.1.smithi124.stderr: 1: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string, std::allocator > const&)+0xd7) [0x56265adc49b5] 2023-08-26T14:05:49.578 INFO:tasks.ceph.cluster2.osd.1.smithi124.stderr: 2: (KernelDevice::_aio_thread()+0x1686) [0x56265b9ee416] 2023-08-26T14:05:49.578 INFO:tasks.ceph.cluster2.osd.1.smithi124.stderr: 3: (KernelDevice::AioCompletionThread::entry()+0x11) [0x56265b9f3ed1] 2023-08-26T14:05:49.579 INFO:tasks.ceph.cluster2.osd.1.smithi124.stderr: 4: /lib64/libpthread.so.0(+0x81cf) [0x7f2a74f7d1cf] 2023-08-26T14:05:49.579 INFO:tasks.ceph.cluster2.osd.1.smithi124.stderr: 5: clone() 2023-08-26T14:05:49.579 INFO:tasks.ceph.cluster2.osd.1.smithi124.stderr: 2023-08-26T14:05:49.579 INFO:tasks.ceph.cluster2.osd.1.smithi124.stderr:*** Caught signal (Aborted) ** 2023-08-26T14:05:49.579 INFO:tasks.ceph.cluster2.osd.1.smithi124.stderr: in thread 7f2a62456700 thread_name:bstore_aio 2023-08-26T14:05:49.580 INFO:tasks.ceph.cluster2.osd.1.smithi124.stderr: ceph version 18.0.0-5706-ge7eb9853 (e7eb9853fddaf661c68f207346131a7e00d8ed2f) reef (dev) 2023-08-26T14:05:49.581 INFO:tasks.ceph.cluster2.osd.1.smithi124.stderr: 1: /lib64/libpthread.so.0(+0x12ce0) [0x7f2a74f87ce0] 2023-08-26T14:05:49.581 INFO:tasks.ceph.cluster2.osd.1.smithi124.stderr: 2: gsignal() 2023-08-26T14:05:49.581 INFO:tasks.ceph.cluster2.osd.1.smithi124.stderr: 3: abort() 2023-08-26T14:05:49.582 INFO:tasks.ceph.cluster2.osd.1.smithi124.stderr: 4: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string, std::allocator > const&)+0x197) [0x56265adc4a75] 2023-08-26T14:05:49.582 INFO:tasks.ceph.cluster2.osd.1.smithi124.stderr: 5: (KernelDevice::_aio_thread()+0x1686) [0x56265b9ee416] 2023-08-26T14:05:49.582 INFO:tasks.ceph.cluster2.osd.1.smithi124.stderr: 6: (KernelDevice::AioCompletionThread::entry()+0x11) [0x56265b9f3ed1] 2023-08-26T14:05:49.583 INFO:tasks.ceph.cluster2.osd.1.smithi124.stderr: 7: /lib64/libpthread.so.0(+0x81cf) [0x7f2a74f7d1cf] 2023-08-26T14:05:49.583 INFO:tasks.ceph.cluster2.osd.1.smithi124.stderr: 8: clone() 2023-08-26T14:05:49.583 INFO:tasks.ceph.cluster2.osd.1.smithi124.stderr:2023-08-26T14:05:49.577+0000 7f2a62456700 -1 *** Caught signal (Aborted) ** 2023-08-26T14:05:49.584 INFO:tasks.ceph.cluster2.osd.1.smithi124.stderr: in thread 7f2a62456700 thread_name:bstore_aio 2023-08-26T14:05:49.584 INFO:tasks.ceph.cluster2.osd.1.smithi124.stderr: 2023-08-26T14:05:49.584 INFO:tasks.ceph.cluster2.osd.1.smithi124.stderr: ceph version 18.0.0-5706-ge7eb9853 (e7eb9853fddaf661c68f207346131a7e00d8ed2f) reef (dev) 2023-08-26T14:05:49.585 INFO:tasks.ceph.cluster2.osd.1.smithi124.stderr: 1: /lib64/libpthread.so.0(+0x12ce0) [0x7f2a74f87ce0] 2023-08-26T14:05:49.585 INFO:tasks.ceph.cluster2.osd.1.smithi124.stderr: 2: gsignal() 2023-08-26T14:05:49.586 INFO:tasks.ceph.cluster2.osd.1.smithi124.stderr: 3: abort() 2023-08-26T14:05:49.586 INFO:tasks.ceph.cluster2.osd.1.smithi124.stderr: 4: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string, std::allocator > const&)+0x197) [0x56265adc4a75] 2023-08-26T14:05:49.586 INFO:tasks.ceph.cluster2.osd.1.smithi124.stderr: 5: (KernelDevice::_aio_thread()+0x1686) [0x56265b9ee416] 2023-08-26T14:05:49.587 INFO:tasks.ceph.cluster2.osd.1.smithi124.stderr: 6: (KernelDevice::AioCompletionThread::entry()+0x11) [0x56265b9f3ed1] 2023-08-26T14:05:49.587 INFO:tasks.ceph.cluster2.osd.1.smithi124.stderr: 7: /lib64/libpthread.so.0(+0x81cf) [0x7f2a74f7d1cf] 2023-08-26T14:05:49.587 INFO:tasks.ceph.cluster2.osd.1.smithi124.stderr: 8: clone() 2023-08-26T14:05:49.587 INFO:tasks.ceph.cluster2.osd.1.smithi124.stderr: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2023-08-26T14:05:49.588 INFO:tasks.ceph.cluster2.osd.1.smithi124.stderr: 2023-08-26T14:05:49.597 INFO:tasks.ceph.cluster2.osd.1.smithi124.stderr: -153> 2023-08-26T14:05:49.573+0000 7f2a62456700 -1 bdev(0x56265f2e0000 /var/lib/ceph/osd/cluster2-1/block) _aio_thread got r=-28 ((28) No space left on device) 2023-08-26T14:05:49.598 INFO:tasks.ceph.cluster2.osd.1.smithi124.stderr: -140> 2023-08-26T14:05:49.574+0000 7f2a62456700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/18.0.0-5706-ge7eb9853/rpm/el8/BUILD/ceph-18.0.0-5706-ge7eb9853/src/blk/kernel/KernelDevice.cc: In function 'void KernelDevice::_aio_thread()' thread 7f2a62456700 time 2023-08-26T14:05:49.573990+0000 2023-08-26T14:05:49.598 INFO:tasks.ceph.cluster2.osd.1.smithi124.stderr:/home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/18.0.0-5706-ge7eb9853/rpm/el8/BUILD/ceph-18.0.0-5706-ge7eb9853/src/blk/kernel/KernelDevice.cc: 633: ceph_abort_msg("Unexpected IO error. This may suggest a hardware issue. Please check your kernel log!") 2023-08-26T14:05:49.598 INFO:tasks.ceph.cluster2.osd.1.smithi124.stderr: 2023-08-26T14:05:49.598 INFO:tasks.ceph.cluster2.osd.1.smithi124.stderr: ceph version 18.0.0-5706-ge7eb9853 (e7eb9853fddaf661c68f207346131a7e00d8ed2f) reef (dev) 2023-08-26T14:05:49.599 INFO:tasks.ceph.cluster2.osd.1.smithi124.stderr: 1: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string, std::allocator > const&)+0xd7) [0x56265adc49b5] 2023-08-26T14:05:49.599 INFO:tasks.ceph.cluster2.osd.1.smithi124.stderr: 2: (KernelDevice::_aio_thread()+0x1686) [0x56265b9ee416] 2023-08-26T14:05:49.599 INFO:tasks.ceph.cluster2.osd.1.smithi124.stderr: 3: (KernelDevice::AioCompletionThread::entry()+0x11) [0x56265b9f3ed1] 2023-08-26T14:05:49.599 INFO:tasks.ceph.cluster2.osd.1.smithi124.stderr: 4: /lib64/libpthread.so.0(+0x81cf) [0x7f2a74f7d1cf] 2023-08-26T14:05:49.599 INFO:tasks.ceph.cluster2.osd.1.smithi124.stderr: 5: clone() 2023-08-26T14:05:49.600 INFO:tasks.ceph.cluster2.osd.1.smithi124.stderr: 2023-08-26T14:05:49.600 INFO:tasks.ceph.cluster2.osd.1.smithi124.stderr: 0> 2023-08-26T14:05:49.577+0000 7f2a62456700 -1 *** Caught signal (Aborted) ** 2023-08-26T14:05:49.600 INFO:tasks.ceph.cluster2.osd.1.smithi124.stderr: in thread 7f2a62456700 thread_name:bstore_aio 2023-08-26T14:05:49.600 INFO:tasks.ceph.cluster2.osd.1.smithi124.stderr: 2023-08-26T14:05:49.601 INFO:tasks.ceph.cluster2.osd.1.smithi124.stderr: ceph version 18.0.0-5706-ge7eb9853 (e7eb9853fddaf661c68f207346131a7e00d8ed2f) reef (dev) 2023-08-26T14:05:49.601 INFO:tasks.ceph.cluster2.osd.1.smithi124.stderr: 1: /lib64/libpthread.so.0(+0x12ce0) [0x7f2a74f87ce0] 2023-08-26T14:05:49.601 INFO:tasks.ceph.cluster2.osd.1.smithi124.stderr: 2: gsignal() 2023-08-26T14:05:49.601 INFO:tasks.ceph.cluster2.osd.1.smithi124.stderr: 3: abort() 2023-08-26T14:05:49.601 INFO:tasks.ceph.cluster2.osd.1.smithi124.stderr: 4: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string, std::allocator > const&)+0x197) [0x56265adc4a75] 2023-08-26T14:05:49.601 INFO:tasks.ceph.cluster2.osd.1.smithi124.stderr: 5: (KernelDevice::_aio_thread()+0x1686) [0x56265b9ee416] 2023-08-26T14:05:49.602 INFO:tasks.ceph.cluster2.osd.1.smithi124.stderr: 6: (KernelDevice::AioCompletionThread::entry()+0x11) [0x56265b9f3ed1] 2023-08-26T14:05:49.602 INFO:tasks.ceph.cluster2.osd.1.smithi124.stderr: 7: /lib64/libpthread.so.0(+0x81cf) [0x7f2a74f7d1cf] 2023-08-26T14:05:49.602 INFO:tasks.ceph.cluster2.osd.1.smithi124.stderr: 8: clone() 2023-08-26T14:05:49.602 INFO:tasks.ceph.cluster2.osd.1.smithi124.stderr: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2023-08-26T14:05:49.602 INFO:tasks.ceph.cluster2.osd.1.smithi124.stderr: 2023-08-26T14:05:49.608 INFO:tasks.ceph.cluster2.osd.1.smithi124.stderr: -1154> 2023-08-26T14:05:49.573+0000 7f2a62456700 -1 bdev(0x56265f2e0000 /var/lib/ceph/osd/cluster2-1/block) _aio_thread got r=-28 ((28) No space left on device) 2023-08-26T14:05:49.609 INFO:tasks.ceph.cluster2.osd.1.smithi124.stderr: -1141> 2023-08-26T14:05:49.574+0000 7f2a62456700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/18.0.0-5706-ge7eb9853/rpm/el8/BUILD/ceph-18.0.0-5706-ge7eb9853/src/blk/kernel/KernelDevice.cc: In function 'void KernelDevice::_aio_thread()' thread 7f2a62456700 time 2023-08-26T14:05:49.573990+0000 2023-08-26T14:05:49.609 INFO:tasks.ceph.cluster2.osd.1.smithi124.stderr:/home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/18.0.0-5706-ge7eb9853/rpm/el8/BUILD/ceph-18.0.0-5706-ge7eb9853/src/blk/kernel/KernelDevice.cc: 633: ceph_abort_msg("Unexpected IO error. This may suggest a hardware issue. Please check your kernel log!") 2023-08-26T14:05:49.609 INFO:tasks.ceph.cluster2.osd.1.smithi124.stderr: 2023-08-26T14:05:49.610 INFO:tasks.ceph.cluster2.osd.1.smithi124.stderr: ceph version 18.0.0-5706-ge7eb9853 (e7eb9853fddaf661c68f207346131a7e00d8ed2f) reef (dev) 2023-08-26T14:05:49.610 INFO:tasks.ceph.cluster2.osd.1.smithi124.stderr: 1: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string, std::allocator > const&)+0xd7) [0x56265adc49b5] 2023-08-26T14:05:49.610 INFO:tasks.ceph.cluster2.osd.1.smithi124.stderr: 2: (KernelDevice::_aio_thread()+0x1686) [0x56265b9ee416] 2023-08-26T14:05:49.611 INFO:tasks.ceph.cluster2.osd.1.smithi124.stderr: 3: (KernelDevice::AioCompletionThread::entry()+0x11) [0x56265b9f3ed1] 2023-08-26T14:05:49.611 INFO:tasks.ceph.cluster2.osd.1.smithi124.stderr: 4: /lib64/libpthread.so.0(+0x81cf) [0x7f2a74f7d1cf] 2023-08-26T14:05:49.612 INFO:tasks.ceph.cluster2.osd.1.smithi124.stderr: 5: clone() 2023-08-26T14:05:49.613 INFO:tasks.ceph.cluster2.osd.1.smithi124.stderr: 2023-08-26T14:05:49.613 INFO:tasks.ceph.cluster2.osd.1.smithi124.stderr: -1001> 2023-08-26T14:05:49.577+0000 7f2a62456700 -1 *** Caught signal (Aborted) ** 2023-08-26T14:05:49.613 INFO:tasks.ceph.cluster2.osd.1.smithi124.stderr: in thread 7f2a62456700 thread_name:bstore_aio 2023-08-26T14:05:49.614 INFO:tasks.ceph.cluster2.osd.1.smithi124.stderr: 2023-08-26T14:05:49.614 INFO:tasks.ceph.cluster2.osd.1.smithi124.stderr: ceph version 18.0.0-5706-ge7eb9853 (e7eb9853fddaf661c68f207346131a7e00d8ed2f) reef (dev) 2023-08-26T14:05:49.614 INFO:tasks.ceph.cluster2.osd.1.smithi124.stderr: 1: /lib64/libpthread.so.0(+0x12ce0) [0x7f2a74f87ce0] 2023-08-26T14:05:49.614 INFO:tasks.ceph.cluster2.osd.1.smithi124.stderr: 2: gsignal() 2023-08-26T14:05:49.614 INFO:tasks.ceph.cluster2.osd.1.smithi124.stderr: 3: abort() 2023-08-26T14:05:49.615 INFO:tasks.ceph.cluster2.osd.1.smithi124.stderr: 4: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string, std::allocator > const&)+0x197) [0x56265adc4a75] 2023-08-26T14:05:49.615 INFO:tasks.ceph.cluster2.osd.1.smithi124.stderr: 5: (KernelDevice::_aio_thread()+0x1686) [0x56265b9ee416] 2023-08-26T14:05:49.615 INFO:tasks.ceph.cluster2.osd.1.smithi124.stderr: 6: (KernelDevice::AioCompletionThread::entry()+0x11) [0x56265b9f3ed1] 2023-08-26T14:05:49.615 INFO:tasks.ceph.cluster2.osd.1.smithi124.stderr: 7: /lib64/libpthread.so.0(+0x81cf) [0x7f2a74f7d1cf] 2023-08-26T14:05:49.615 INFO:tasks.ceph.cluster2.osd.1.smithi124.stderr: 8: clone() 2023-08-26T14:05:49.615 INFO:tasks.ceph.cluster2.osd.1.smithi124.stderr: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2023-08-26T14:05:49.616 INFO:tasks.ceph.cluster2.osd.1.smithi124.stderr: 2023-08-26T14:05:49.778 INFO:tasks.ceph.cluster2.osd.1.smithi124.stderr:daemon-helper: command failed with exit status 5 2023-08-26T14:05:52.978 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.0 is failed for ~211s 2023-08-26T14:05:52.978 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.1 is failed for ~0s 2023-08-26T14:05:58.282 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.0 is failed for ~216s 2023-08-26T14:05:58.282 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.1 is failed for ~5s 2023-08-26T14:05:58.506 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ error_code=124 2023-08-26T14:05:58.506 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ set -e 2023-08-26T14:05:58.507 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ '[' 124 -eq 124 ']' 2023-08-26T14:05:58.507 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ return 0 2023-08-26T14:05:58.507 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ wait_for_image_present cluster1 mirror image_7 present 2023-08-26T14:05:58.507 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster1 2023-08-26T14:05:58.508 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T14:05:58.508 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=image_7 2023-08-26T14:05:58.508 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local state=present 2023-08-26T14:05:58.508 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image_id= 2023-08-26T14:05:58.508 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local s 2023-08-26T14:05:58.508 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ test -n '' 2023-08-26T14:05:58.509 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ get_image_id cluster1 mirror image_7 2023-08-26T14:05:58.509 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local cluster=cluster1 2023-08-26T14:05:58.509 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local pool=mirror 2023-08-26T14:05:58.510 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local image=image_7 2023-08-26T14:05:58.510 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ rbd --cluster=cluster1 info mirror/image_7 2023-08-26T14:05:58.510 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2023-08-26T14:05:58.542 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:rbd: error opening image image_7: (2) No such file or directory 2023-08-26T14:05:58.545 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ image_id= 2023-08-26T14:05:58.545 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2023-08-26T14:05:58.546 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ sleep 0.1 2023-08-26T14:05:58.647 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ test_image_present cluster1 mirror image_7 present '' 2023-08-26T14:05:58.648 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster1 2023-08-26T14:05:58.648 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T14:05:58.648 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=image_7 2023-08-26T14:05:58.648 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local test_state=present 2023-08-26T14:05:58.648 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image_id= 2023-08-26T14:05:58.649 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local current_state=deleted 2023-08-26T14:05:58.649 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local current_image_id 2023-08-26T14:05:58.649 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ get_image_id cluster1 mirror image_7 2023-08-26T14:05:58.649 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local cluster=cluster1 2023-08-26T14:05:58.649 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local pool=mirror 2023-08-26T14:05:58.650 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local image=image_7 2023-08-26T14:05:58.650 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ rbd --cluster=cluster1 info mirror/image_7 2023-08-26T14:05:58.650 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2023-08-26T14:05:58.680 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:rbd: error opening image image_7: (2) No such file or directory 2023-08-26T14:05:58.684 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ current_image_id= 2023-08-26T14:05:58.684 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ test -n '' 2023-08-26T14:05:58.684 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ test present = deleted 2023-08-26T14:05:58.685 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2023-08-26T14:05:58.685 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ sleep 1 2023-08-26T14:05:59.686 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ test_image_present cluster1 mirror image_7 present '' 2023-08-26T14:05:59.686 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster1 2023-08-26T14:05:59.686 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T14:05:59.687 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=image_7 2023-08-26T14:05:59.687 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local test_state=present 2023-08-26T14:05:59.687 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image_id= 2023-08-26T14:05:59.687 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local current_state=deleted 2023-08-26T14:05:59.687 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local current_image_id 2023-08-26T14:05:59.688 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ get_image_id cluster1 mirror image_7 2023-08-26T14:05:59.688 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local cluster=cluster1 2023-08-26T14:05:59.688 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local pool=mirror 2023-08-26T14:05:59.688 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local image=image_7 2023-08-26T14:05:59.688 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ rbd --cluster=cluster1 info mirror/image_7 2023-08-26T14:05:59.689 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2023-08-26T14:05:59.721 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:rbd: error opening image image_7: (2) No such file or directory 2023-08-26T14:05:59.725 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ current_image_id= 2023-08-26T14:05:59.725 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ test -n '' 2023-08-26T14:05:59.725 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ test present = deleted 2023-08-26T14:05:59.726 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2023-08-26T14:05:59.726 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ sleep 2 2023-08-26T14:06:01.727 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ test_image_present cluster1 mirror image_7 present '' 2023-08-26T14:06:01.728 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster1 2023-08-26T14:06:01.728 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T14:06:01.728 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=image_7 2023-08-26T14:06:01.728 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local test_state=present 2023-08-26T14:06:01.728 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image_id= 2023-08-26T14:06:01.729 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local current_state=deleted 2023-08-26T14:06:01.729 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local current_image_id 2023-08-26T14:06:01.729 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ get_image_id cluster1 mirror image_7 2023-08-26T14:06:01.729 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local cluster=cluster1 2023-08-26T14:06:01.729 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local pool=mirror 2023-08-26T14:06:01.729 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local image=image_7 2023-08-26T14:06:01.730 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ rbd --cluster=cluster1 info mirror/image_7 2023-08-26T14:06:01.730 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2023-08-26T14:06:01.765 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:rbd: error opening image image_7: (2) No such file or directory 2023-08-26T14:06:01.768 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ current_image_id= 2023-08-26T14:06:01.768 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ test -n '' 2023-08-26T14:06:01.769 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ test present = deleted 2023-08-26T14:06:01.769 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2023-08-26T14:06:01.769 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ sleep 4 2023-08-26T14:06:03.585 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.0 is failed for ~221s 2023-08-26T14:06:03.585 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.1 is failed for ~11s 2023-08-26T14:06:05.770 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ test_image_present cluster1 mirror image_7 present '' 2023-08-26T14:06:05.770 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster1 2023-08-26T14:06:05.770 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T14:06:05.771 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=image_7 2023-08-26T14:06:05.771 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local test_state=present 2023-08-26T14:06:05.771 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image_id= 2023-08-26T14:06:05.771 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local current_state=deleted 2023-08-26T14:06:05.771 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local current_image_id 2023-08-26T14:06:05.772 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ get_image_id cluster1 mirror image_7 2023-08-26T14:06:05.772 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local cluster=cluster1 2023-08-26T14:06:05.772 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local pool=mirror 2023-08-26T14:06:05.772 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local image=image_7 2023-08-26T14:06:05.772 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ rbd --cluster=cluster1 info mirror/image_7 2023-08-26T14:06:05.773 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2023-08-26T14:06:05.805 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:rbd: error opening image image_7: (2) No such file or directory 2023-08-26T14:06:05.809 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ current_image_id= 2023-08-26T14:06:05.809 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ test -n '' 2023-08-26T14:06:05.809 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ test present = deleted 2023-08-26T14:06:05.809 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2023-08-26T14:06:05.809 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ sleep 8 2023-08-26T14:06:08.888 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.0 is failed for ~227s 2023-08-26T14:06:08.888 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.1 is failed for ~16s 2023-08-26T14:06:13.810 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ test_image_present cluster1 mirror image_7 present '' 2023-08-26T14:06:13.811 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster1 2023-08-26T14:06:13.811 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T14:06:13.811 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=image_7 2023-08-26T14:06:13.811 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local test_state=present 2023-08-26T14:06:13.812 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image_id= 2023-08-26T14:06:13.812 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local current_state=deleted 2023-08-26T14:06:13.812 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local current_image_id 2023-08-26T14:06:13.812 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ get_image_id cluster1 mirror image_7 2023-08-26T14:06:13.812 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local cluster=cluster1 2023-08-26T14:06:13.813 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local pool=mirror 2023-08-26T14:06:13.813 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local image=image_7 2023-08-26T14:06:13.813 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ rbd --cluster=cluster1 info mirror/image_7 2023-08-26T14:06:13.813 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2023-08-26T14:06:13.845 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:rbd: error opening image image_7: (2) No such file or directory 2023-08-26T14:06:13.849 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ current_image_id= 2023-08-26T14:06:13.849 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ test -n '' 2023-08-26T14:06:13.849 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ test present = deleted 2023-08-26T14:06:13.849 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2023-08-26T14:06:13.850 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ sleep 8 2023-08-26T14:06:14.191 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.0 is failed for ~232s 2023-08-26T14:06:14.191 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.1 is failed for ~21s 2023-08-26T14:06:19.494 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.0 is failed for ~237s 2023-08-26T14:06:19.495 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.1 is failed for ~27s 2023-08-26T14:06:21.851 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ test_image_present cluster1 mirror image_7 present '' 2023-08-26T14:06:21.851 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster1 2023-08-26T14:06:21.851 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T14:06:21.852 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=image_7 2023-08-26T14:06:21.852 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local test_state=present 2023-08-26T14:06:21.852 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image_id= 2023-08-26T14:06:21.852 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local current_state=deleted 2023-08-26T14:06:21.853 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local current_image_id 2023-08-26T14:06:21.853 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ get_image_id cluster1 mirror image_7 2023-08-26T14:06:21.853 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local cluster=cluster1 2023-08-26T14:06:21.853 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local pool=mirror 2023-08-26T14:06:21.853 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local image=image_7 2023-08-26T14:06:21.853 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ rbd --cluster=cluster1 info mirror/image_7 2023-08-26T14:06:21.854 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2023-08-26T14:06:21.888 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:rbd: error opening image image_7: (2) No such file or directory 2023-08-26T14:06:21.892 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ current_image_id= 2023-08-26T14:06:21.892 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ test -n '' 2023-08-26T14:06:21.892 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ test present = deleted 2023-08-26T14:06:21.892 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2023-08-26T14:06:21.892 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ sleep 8 2023-08-26T14:06:24.798 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.0 is failed for ~243s 2023-08-26T14:06:24.799 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.1 is failed for ~32s 2023-08-26T14:06:29.894 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ test_image_present cluster1 mirror image_7 present '' 2023-08-26T14:06:29.894 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster1 2023-08-26T14:06:29.894 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T14:06:29.894 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=image_7 2023-08-26T14:06:29.895 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local test_state=present 2023-08-26T14:06:29.895 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image_id= 2023-08-26T14:06:29.895 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local current_state=deleted 2023-08-26T14:06:29.895 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local current_image_id 2023-08-26T14:06:29.895 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ get_image_id cluster1 mirror image_7 2023-08-26T14:06:29.896 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local cluster=cluster1 2023-08-26T14:06:29.896 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local pool=mirror 2023-08-26T14:06:29.896 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local image=image_7 2023-08-26T14:06:29.896 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ rbd --cluster=cluster1 info mirror/image_7 2023-08-26T14:06:29.896 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2023-08-26T14:06:29.931 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:rbd: error opening image image_7: (2) No such file or directory 2023-08-26T14:06:29.934 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ current_image_id= 2023-08-26T14:06:29.935 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ test -n '' 2023-08-26T14:06:29.935 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ test present = deleted 2023-08-26T14:06:29.935 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2023-08-26T14:06:29.935 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ sleep 8 2023-08-26T14:06:30.102 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.0 is failed for ~248s 2023-08-26T14:06:30.102 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.1 is failed for ~37s 2023-08-26T14:06:35.406 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.0 is failed for ~253s 2023-08-26T14:06:35.406 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.1 is failed for ~42s 2023-08-26T14:06:37.936 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ test_image_present cluster1 mirror image_7 present '' 2023-08-26T14:06:37.937 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster1 2023-08-26T14:06:37.937 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T14:06:37.937 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=image_7 2023-08-26T14:06:37.937 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local test_state=present 2023-08-26T14:06:37.937 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image_id= 2023-08-26T14:06:37.938 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local current_state=deleted 2023-08-26T14:06:37.938 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local current_image_id 2023-08-26T14:06:37.938 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ get_image_id cluster1 mirror image_7 2023-08-26T14:06:37.938 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local cluster=cluster1 2023-08-26T14:06:37.938 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local pool=mirror 2023-08-26T14:06:37.939 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local image=image_7 2023-08-26T14:06:37.939 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ rbd --cluster=cluster1 info mirror/image_7 2023-08-26T14:06:37.939 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2023-08-26T14:06:37.972 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:rbd: error opening image image_7: (2) No such file or directory 2023-08-26T14:06:37.975 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ current_image_id= 2023-08-26T14:06:37.976 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ test -n '' 2023-08-26T14:06:37.976 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ test present = deleted 2023-08-26T14:06:37.976 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2023-08-26T14:06:37.976 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ sleep 8 2023-08-26T14:06:40.709 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.0 is failed for ~258s 2023-08-26T14:06:40.710 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.1 is failed for ~48s 2023-08-26T14:06:45.977 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ test_image_present cluster1 mirror image_7 present '' 2023-08-26T14:06:45.978 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster1 2023-08-26T14:06:45.978 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T14:06:45.978 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=image_7 2023-08-26T14:06:45.978 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local test_state=present 2023-08-26T14:06:45.978 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image_id= 2023-08-26T14:06:45.979 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local current_state=deleted 2023-08-26T14:06:45.979 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local current_image_id 2023-08-26T14:06:45.979 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ get_image_id cluster1 mirror image_7 2023-08-26T14:06:45.979 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local cluster=cluster1 2023-08-26T14:06:45.979 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local pool=mirror 2023-08-26T14:06:45.980 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local image=image_7 2023-08-26T14:06:45.980 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ rbd --cluster=cluster1 info mirror/image_7 2023-08-26T14:06:45.980 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2023-08-26T14:06:46.013 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.0 is failed for ~264s 2023-08-26T14:06:46.013 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.1 is failed for ~53s 2023-08-26T14:06:46.014 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:rbd: error opening image image_7: (2) No such file or directory 2023-08-26T14:06:46.017 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ current_image_id= 2023-08-26T14:06:46.017 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ test -n '' 2023-08-26T14:06:46.017 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ test present = deleted 2023-08-26T14:06:46.018 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2023-08-26T14:06:46.018 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ sleep 8 2023-08-26T14:06:51.316 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.0 is failed for ~269s 2023-08-26T14:06:51.317 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.1 is failed for ~58s 2023-08-26T14:06:54.019 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ test_image_present cluster1 mirror image_7 present '' 2023-08-26T14:06:54.019 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster1 2023-08-26T14:06:54.020 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T14:06:54.020 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=image_7 2023-08-26T14:06:54.020 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local test_state=present 2023-08-26T14:06:54.020 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image_id= 2023-08-26T14:06:54.020 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local current_state=deleted 2023-08-26T14:06:54.021 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local current_image_id 2023-08-26T14:06:54.021 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ get_image_id cluster1 mirror image_7 2023-08-26T14:06:54.021 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local cluster=cluster1 2023-08-26T14:06:54.021 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local pool=mirror 2023-08-26T14:06:54.021 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local image=image_7 2023-08-26T14:06:54.022 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ rbd --cluster=cluster1 info mirror/image_7 2023-08-26T14:06:54.022 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2023-08-26T14:06:54.054 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:rbd: error opening image image_7: (2) No such file or directory 2023-08-26T14:06:54.057 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ current_image_id= 2023-08-26T14:06:54.058 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ test -n '' 2023-08-26T14:06:54.058 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ test present = deleted 2023-08-26T14:06:54.058 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2023-08-26T14:06:54.058 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ sleep 8 2023-08-26T14:06:56.620 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.0 is failed for ~274s 2023-08-26T14:06:56.620 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.1 is failed for ~64s 2023-08-26T14:06:57.784 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.0 with skip (0.830092875256504) > weight (0.25) 2023-08-26T14:06:57.785 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.1 with skip (0.9710651827038733) > weight (0.25) 2023-08-26T14:06:57.785 INFO:tasks.rbd_mirror_thrash:kill cluster1.client.mirror.2 2023-08-26T14:06:57.785 INFO:tasks.rbd_mirror.cluster1.client.mirror.2:Sent signal 15 2023-08-26T14:06:57.786 INFO:tasks.rbd_mirror_thrash:waiting for 8 secs before reviving daemons 2023-08-26T14:06:57.786 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi124.stderr:2023-08-26T14:06:57.785+0000 7f733997f700 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.2 (PID: 130412) UID: 1000 2023-08-26T14:07:01.924 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.0 is failed for ~280s 2023-08-26T14:07:01.924 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.1 is failed for ~69s 2023-08-26T14:07:02.059 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ test_image_present cluster1 mirror image_7 present '' 2023-08-26T14:07:02.060 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster1 2023-08-26T14:07:02.060 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T14:07:02.060 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=image_7 2023-08-26T14:07:02.060 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local test_state=present 2023-08-26T14:07:02.061 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image_id= 2023-08-26T14:07:02.061 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local current_state=deleted 2023-08-26T14:07:02.061 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local current_image_id 2023-08-26T14:07:02.061 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ get_image_id cluster1 mirror image_7 2023-08-26T14:07:02.061 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local cluster=cluster1 2023-08-26T14:07:02.062 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local pool=mirror 2023-08-26T14:07:02.062 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local image=image_7 2023-08-26T14:07:02.062 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ rbd --cluster=cluster1 info mirror/image_7 2023-08-26T14:07:02.062 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2023-08-26T14:07:02.096 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:rbd: error opening image image_7: (2) No such file or directory 2023-08-26T14:07:02.100 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ current_image_id= 2023-08-26T14:07:02.100 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ test -n '' 2023-08-26T14:07:02.100 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ test present = deleted 2023-08-26T14:07:02.100 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2023-08-26T14:07:02.101 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ sleep 8 2023-08-26T14:07:02.567 INFO:tasks.rbd_mirror_thrash:kill cluster2.client.mirror.0 2023-08-26T14:07:02.568 INFO:tasks.rbd_mirror.cluster2.client.mirror.0:Sent signal 15 2023-08-26T14:07:02.568 INFO:tasks.rbd_mirror_thrash:waiting for 3 secs before reviving daemons 2023-08-26T14:07:02.568 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi124.stderr:2023-08-26T14:07:02.568+0000 7fc0ca944700 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.0 (PID: 131554) UID: 1000 2023-08-26T14:07:05.569 INFO:tasks.rbd_mirror_thrash:waiting for cluster2.client.mirror.0 2023-08-26T14:07:05.569 INFO:teuthology.orchestra.run:waiting for 600 2023-08-26T14:07:05.787 INFO:tasks.rbd_mirror_thrash:waiting for cluster1.client.mirror.2 2023-08-26T14:07:05.787 INFO:teuthology.orchestra.run:waiting for 600 2023-08-26T14:07:07.227 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.0 is failed for ~285s 2023-08-26T14:07:07.228 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.1 is failed for ~74s 2023-08-26T14:07:10.102 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ test_image_present cluster1 mirror image_7 present '' 2023-08-26T14:07:10.102 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster1 2023-08-26T14:07:10.102 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T14:07:10.103 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=image_7 2023-08-26T14:07:10.103 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local test_state=present 2023-08-26T14:07:10.103 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image_id= 2023-08-26T14:07:10.103 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local current_state=deleted 2023-08-26T14:07:10.103 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local current_image_id 2023-08-26T14:07:10.104 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ get_image_id cluster1 mirror image_7 2023-08-26T14:07:10.104 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local cluster=cluster1 2023-08-26T14:07:10.104 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local pool=mirror 2023-08-26T14:07:10.104 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local image=image_7 2023-08-26T14:07:10.104 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ rbd --cluster=cluster1 info mirror/image_7 2023-08-26T14:07:10.104 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2023-08-26T14:07:10.139 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:rbd: error opening image image_7: (2) No such file or directory 2023-08-26T14:07:10.143 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ current_image_id= 2023-08-26T14:07:10.143 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ test -n '' 2023-08-26T14:07:10.143 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ test present = deleted 2023-08-26T14:07:10.143 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2023-08-26T14:07:10.143 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ sleep 16 2023-08-26T14:07:12.531 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.0 is failed for ~290s 2023-08-26T14:07:12.531 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.1 is failed for ~80s 2023-08-26T14:07:17.835 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.0 is failed for ~296s 2023-08-26T14:07:17.835 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.1 is failed for ~85s 2023-08-26T14:07:23.138 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.0 is failed for ~301s 2023-08-26T14:07:23.138 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.1 is failed for ~90s 2023-08-26T14:07:23.138 INFO:tasks.daemonwatchdog.daemon_watchdog:BARK! unmounting mounts and killing all daemons 2023-08-26T14:07:23.441 INFO:tasks.ceph.cluster2.osd.2:Sent signal 15 2023-08-26T14:07:23.442 INFO:tasks.ceph.cluster2.mon.a:Sent signal 15 2023-08-26T14:07:23.442 INFO:tasks.ceph.cluster2.mgr.x:Sent signal 15 2023-08-26T14:07:23.443 INFO:tasks.ceph.cluster2.osd.2.smithi124.stderr:2023-08-26T14:07:23.440+0000 7fb79eb4a700 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster cluster2 -i 2 (PID: 99890) UID: 0 2023-08-26T14:07:23.443 INFO:tasks.ceph.cluster2.osd.2.smithi124.stderr:2023-08-26T14:07:23.440+0000 7fb79eb4a700 -1 osd.2 439 *** Got signal Terminated *** 2023-08-26T14:07:23.443 INFO:tasks.ceph.cluster2.osd.2.smithi124.stderr:2023-08-26T14:07:23.440+0000 7fb79eb4a700 -1 osd.2 439 *** Immediate shutdown (osd_fast_shutdown=true) *** 2023-08-26T14:07:23.444 INFO:tasks.ceph.cluster2.mon.a.smithi124.stderr:2023-08-26T14:07:23.440+0000 7f154e50d700 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper kill ceph-mon -f --cluster cluster2 -i a (PID: 99545) UID: 0 2023-08-26T14:07:23.444 INFO:tasks.ceph.cluster2.mon.a.smithi124.stderr:2023-08-26T14:07:23.440+0000 7f154e50d700 -1 mon.a@0(leader) e1 *** Got Signal Terminated *** 2023-08-26T14:07:23.445 INFO:tasks.ceph.cluster2.mgr.x.smithi031.stderr:2023-08-26T14:07:23.443+0000 7f86c84a8700 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper kill ceph-mgr -f --cluster cluster2 -i x (PID: 105110) UID: 0 2023-08-26T14:07:23.445 INFO:tasks.ceph.cluster2.mgr.x.smithi031.stderr:2023-08-26T14:07:23.443+0000 7f86c84a8700 -1 mgr handle_mgr_signal *** Got signal Terminated *** 2023-08-26T14:07:26.145 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ test_image_present cluster1 mirror image_7 present '' 2023-08-26T14:07:26.145 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster1 2023-08-26T14:07:26.145 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T14:07:26.146 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=image_7 2023-08-26T14:07:26.146 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local test_state=present 2023-08-26T14:07:26.146 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image_id= 2023-08-26T14:07:26.146 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local current_state=deleted 2023-08-26T14:07:26.146 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local current_image_id 2023-08-26T14:07:26.147 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ get_image_id cluster1 mirror image_7 2023-08-26T14:07:26.147 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local cluster=cluster1 2023-08-26T14:07:26.147 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local pool=mirror 2023-08-26T14:07:26.147 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local image=image_7 2023-08-26T14:07:26.147 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ rbd --cluster=cluster1 info mirror/image_7 2023-08-26T14:07:26.147 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2023-08-26T14:07:26.180 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:rbd: error opening image image_7: (2) No such file or directory 2023-08-26T14:07:26.183 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ current_image_id= 2023-08-26T14:07:26.184 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ test -n '' 2023-08-26T14:07:26.184 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ test present = deleted 2023-08-26T14:07:26.184 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2023-08-26T14:07:26.184 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ sleep 16 2023-08-26T14:07:42.186 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ test_image_present cluster1 mirror image_7 present '' 2023-08-26T14:07:42.186 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster1 2023-08-26T14:07:42.186 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T14:07:42.186 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=image_7 2023-08-26T14:07:42.187 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local test_state=present 2023-08-26T14:07:42.187 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image_id= 2023-08-26T14:07:42.187 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local current_state=deleted 2023-08-26T14:07:42.187 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local current_image_id 2023-08-26T14:07:42.187 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ get_image_id cluster1 mirror image_7 2023-08-26T14:07:42.188 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local cluster=cluster1 2023-08-26T14:07:42.188 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local pool=mirror 2023-08-26T14:07:42.188 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local image=image_7 2023-08-26T14:07:42.188 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ rbd --cluster=cluster1 info mirror/image_7 2023-08-26T14:07:42.188 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2023-08-26T14:07:42.222 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:rbd: error opening image image_7: (2) No such file or directory 2023-08-26T14:07:42.226 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ current_image_id= 2023-08-26T14:07:42.226 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ test -n '' 2023-08-26T14:07:42.226 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ test present = deleted 2023-08-26T14:07:42.227 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2023-08-26T14:07:42.227 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ sleep 32 2023-08-26T14:08:14.228 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ test_image_present cluster1 mirror image_7 present '' 2023-08-26T14:08:14.228 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster1 2023-08-26T14:08:14.229 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T14:08:14.229 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=image_7 2023-08-26T14:08:14.229 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local test_state=present 2023-08-26T14:08:14.229 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image_id= 2023-08-26T14:08:14.229 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local current_state=deleted 2023-08-26T14:08:14.230 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local current_image_id 2023-08-26T14:08:14.230 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ get_image_id cluster1 mirror image_7 2023-08-26T14:08:14.230 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local cluster=cluster1 2023-08-26T14:08:14.230 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local pool=mirror 2023-08-26T14:08:14.230 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local image=image_7 2023-08-26T14:08:14.231 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ rbd --cluster=cluster1 info mirror/image_7 2023-08-26T14:08:14.231 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2023-08-26T14:08:14.263 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:rbd: error opening image image_7: (2) No such file or directory 2023-08-26T14:08:14.266 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ current_image_id= 2023-08-26T14:08:14.267 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ test -n '' 2023-08-26T14:08:14.267 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ test present = deleted 2023-08-26T14:08:14.267 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2023-08-26T14:08:14.267 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ sleep 32 2023-08-26T14:08:46.268 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ test_image_present cluster1 mirror image_7 present '' 2023-08-26T14:08:46.269 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster1 2023-08-26T14:08:46.269 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pool=mirror 2023-08-26T14:08:46.269 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image=image_7 2023-08-26T14:08:46.270 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local test_state=present 2023-08-26T14:08:46.270 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local image_id= 2023-08-26T14:08:46.270 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local current_state=deleted 2023-08-26T14:08:46.270 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local current_image_id 2023-08-26T14:08:46.270 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ get_image_id cluster1 mirror image_7 2023-08-26T14:08:46.271 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local cluster=cluster1 2023-08-26T14:08:46.271 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local pool=mirror 2023-08-26T14:08:46.271 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local image=image_7 2023-08-26T14:08:46.271 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ rbd --cluster=cluster1 info mirror/image_7 2023-08-26T14:08:46.271 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2023-08-26T14:08:46.305 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:rbd: error opening image image_7: (2) No such file or directory 2023-08-26T14:08:46.309 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ current_image_id= 2023-08-26T14:08:46.309 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ test -n '' 2023-08-26T14:08:46.310 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ test present = deleted 2023-08-26T14:08:46.310 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ return 1 2023-08-26T14:08:46.310 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ cleanup 1 2023-08-26T14:08:46.310 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local error_code=1 2023-08-26T14:08:46.311 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ set +e 2023-08-26T14:08:46.311 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ '[' 1 -ne 0 ']' 2023-08-26T14:08:46.311 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ status 2023-08-26T14:08:46.311 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster daemon image_pool image_ns image 2023-08-26T14:08:46.311 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ for cluster in ${CLUSTER1} ${CLUSTER2} 2023-08-26T14:08:46.312 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ echo 'cluster1 status' 2023-08-26T14:08:46.312 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:cluster1 status 2023-08-26T14:08:46.312 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ CEPH_ARGS= 2023-08-26T14:08:46.313 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ ceph --cluster cluster1 -s 2023-08-26T14:08:46.874 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: cluster: 2023-08-26T14:08:46.874 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: id: f17edf88-b4bf-49db-bf48-f12793acdc0b 2023-08-26T14:08:46.874 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: health: HEALTH_OK 2023-08-26T14:08:46.875 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: 2023-08-26T14:08:46.875 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: services: 2023-08-26T14:08:46.875 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: mon: 1 daemons, quorum a (age 37m) 2023-08-26T14:08:46.875 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: mgr: x(active, since 37m) 2023-08-26T14:08:46.875 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: osd: 3 osds: 3 up (since 37m), 3 in (since 37m) 2023-08-26T14:08:46.876 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: rbd-mirror: 4 daemons active (1 hosts) 2023-08-26T14:08:46.876 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: 2023-08-26T14:08:46.876 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: data: 2023-08-26T14:08:46.876 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: pools: 4 pools, 137 pgs 2023-08-26T14:08:46.876 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: objects: 33 objects, 577 KiB 2023-08-26T14:08:46.876 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: usage: 295 MiB used, 270 GiB / 270 GiB avail 2023-08-26T14:08:46.877 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: pgs: 137 active+clean 2023-08-26T14:08:46.877 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: 2023-08-26T14:08:46.877 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: io: 2023-08-26T14:08:46.877 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: client: 2.3 KiB/s rd, 511 B/s wr, 2 op/s rd, 0 op/s wr 2023-08-26T14:08:46.877 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: 2023-08-26T14:08:46.886 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ CEPH_ARGS= 2023-08-26T14:08:46.887 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ ceph --cluster cluster1 service dump 2023-08-26T14:08:47.291 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:{ 2023-08-26T14:08:47.292 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: "epoch": 44, 2023-08-26T14:08:47.292 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: "modified": "2023-08-26T14:08:38.628055+0000", 2023-08-26T14:08:47.292 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: "services": { 2023-08-26T14:08:47.292 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: "osd": { 2023-08-26T14:08:47.293 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: "daemons": { 2023-08-26T14:08:47.293 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: "summary": "", 2023-08-26T14:08:47.293 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: "0": { 2023-08-26T14:08:47.293 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: "start_epoch": 0, 2023-08-26T14:08:47.293 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: "start_stamp": "0.000000", 2023-08-26T14:08:47.293 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: "gid": 0, 2023-08-26T14:08:47.294 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: "addr": "(unrecognized address family 0)/0", 2023-08-26T14:08:47.294 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: "metadata": {}, 2023-08-26T14:08:47.294 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: "task_status": {} 2023-08-26T14:08:47.294 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: } 2023-08-26T14:08:47.294 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: } 2023-08-26T14:08:47.295 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: }, 2023-08-26T14:08:47.295 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: "rbd-mirror": { 2023-08-26T14:08:47.295 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: "daemons": { 2023-08-26T14:08:47.295 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: "summary": "", 2023-08-26T14:08:47.295 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: "4797": { 2023-08-26T14:08:47.296 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: "start_epoch": 23, 2023-08-26T14:08:47.296 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: "start_stamp": "2023-08-26T13:45:57.832721+0000", 2023-08-26T14:08:47.296 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: "gid": 4797, 2023-08-26T14:08:47.296 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: "addr": "172.21.15.124:0/2111094007", 2023-08-26T14:08:47.296 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: "metadata": { 2023-08-26T14:08:47.296 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: "arch": "x86_64", 2023-08-26T14:08:47.297 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: "ceph_release": "reef", 2023-08-26T14:08:47.297 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: "ceph_version": "ceph version 18.0.0-5706-ge7eb9853 (e7eb9853fddaf661c68f207346131a7e00d8ed2f) reef (dev)", 2023-08-26T14:08:47.297 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: "ceph_version_short": "18.0.0-5706-ge7eb9853", 2023-08-26T14:08:47.297 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: "cpu": "Intel(R) Xeon(R) CPU E5-1620 v4 @ 3.50GHz", 2023-08-26T14:08:47.297 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: "distro": "rhel", 2023-08-26T14:08:47.298 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: "distro_description": "Red Hat Enterprise Linux 8.6 (Ootpa)", 2023-08-26T14:08:47.298 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: "distro_version": "8.6", 2023-08-26T14:08:47.298 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: "hostname": "smithi124", 2023-08-26T14:08:47.298 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: "id": "mirror.1", 2023-08-26T14:08:47.298 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: "instance_id": "4797", 2023-08-26T14:08:47.299 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: "kernel_description": "#1 SMP Fri Apr 15 22:12:19 EDT 2022", 2023-08-26T14:08:47.299 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: "kernel_version": "4.18.0-372.9.1.el8.x86_64", 2023-08-26T14:08:47.299 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: "mem_swap_kb": "0", 2023-08-26T14:08:47.299 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: "mem_total_kb": "32605412", 2023-08-26T14:08:47.299 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: "os": "Linux" 2023-08-26T14:08:47.300 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: }, 2023-08-26T14:08:47.300 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: "task_status": {} 2023-08-26T14:08:47.300 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: }, 2023-08-26T14:08:47.300 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: "5144": { 2023-08-26T14:08:47.300 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: "start_epoch": 30, 2023-08-26T14:08:47.301 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: "start_stamp": "2023-08-26T13:54:09.838868+0000", 2023-08-26T14:08:47.301 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: "gid": 5144, 2023-08-26T14:08:47.301 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: "addr": "172.21.15.124:0/3699195845", 2023-08-26T14:08:47.301 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: "metadata": { 2023-08-26T14:08:47.301 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: "arch": "x86_64", 2023-08-26T14:08:47.302 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: "ceph_release": "reef", 2023-08-26T14:08:47.302 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: "ceph_version": "ceph version 18.0.0-5706-ge7eb9853 (e7eb9853fddaf661c68f207346131a7e00d8ed2f) reef (dev)", 2023-08-26T14:08:47.302 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: "ceph_version_short": "18.0.0-5706-ge7eb9853", 2023-08-26T14:08:47.302 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: "cpu": "Intel(R) Xeon(R) CPU E5-1620 v4 @ 3.50GHz", 2023-08-26T14:08:47.302 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: "distro": "rhel", 2023-08-26T14:08:47.302 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: "distro_description": "Red Hat Enterprise Linux 8.6 (Ootpa)", 2023-08-26T14:08:47.303 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: "distro_version": "8.6", 2023-08-26T14:08:47.303 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: "hostname": "smithi124", 2023-08-26T14:08:47.303 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: "id": "mirror.3", 2023-08-26T14:08:47.303 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: "instance_id": "5144", 2023-08-26T14:08:47.303 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: "kernel_description": "#1 SMP Fri Apr 15 22:12:19 EDT 2022", 2023-08-26T14:08:47.304 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: "kernel_version": "4.18.0-372.9.1.el8.x86_64", 2023-08-26T14:08:47.304 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: "mem_swap_kb": "0", 2023-08-26T14:08:47.304 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: "mem_total_kb": "32605412", 2023-08-26T14:08:47.304 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: "os": "Linux" 2023-08-26T14:08:47.304 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: }, 2023-08-26T14:08:47.305 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: "task_status": {} 2023-08-26T14:08:47.305 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: }, 2023-08-26T14:08:47.305 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: "5530": { 2023-08-26T14:08:47.305 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: "start_epoch": 37, 2023-08-26T14:08:47.305 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: "start_stamp": "2023-08-26T13:57:49.848946+0000", 2023-08-26T14:08:47.306 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: "gid": 5530, 2023-08-26T14:08:47.306 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: "addr": "172.21.15.124:0/4088019104", 2023-08-26T14:08:47.306 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: "metadata": { 2023-08-26T14:08:47.306 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: "arch": "x86_64", 2023-08-26T14:08:47.306 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: "ceph_release": "reef", 2023-08-26T14:08:47.306 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: "ceph_version": "ceph version 18.0.0-5706-ge7eb9853 (e7eb9853fddaf661c68f207346131a7e00d8ed2f) reef (dev)", 2023-08-26T14:08:47.307 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: "ceph_version_short": "18.0.0-5706-ge7eb9853", 2023-08-26T14:08:47.307 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: "cpu": "Intel(R) Xeon(R) CPU E5-1620 v4 @ 3.50GHz", 2023-08-26T14:08:47.307 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: "distro": "rhel", 2023-08-26T14:08:47.307 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: "distro_description": "Red Hat Enterprise Linux 8.6 (Ootpa)", 2023-08-26T14:08:47.307 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: "distro_version": "8.6", 2023-08-26T14:08:47.308 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: "hostname": "smithi124", 2023-08-26T14:08:47.308 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: "id": "mirror.2", 2023-08-26T14:08:47.308 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: "instance_id": "5530", 2023-08-26T14:08:47.308 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: "kernel_description": "#1 SMP Fri Apr 15 22:12:19 EDT 2022", 2023-08-26T14:08:47.308 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: "kernel_version": "4.18.0-372.9.1.el8.x86_64", 2023-08-26T14:08:47.309 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: "mem_swap_kb": "0", 2023-08-26T14:08:47.309 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: "mem_total_kb": "32605412", 2023-08-26T14:08:47.309 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: "os": "Linux" 2023-08-26T14:08:47.309 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: }, 2023-08-26T14:08:47.309 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: "task_status": {} 2023-08-26T14:08:47.310 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: }, 2023-08-26T14:08:47.310 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: "6118": { 2023-08-26T14:08:47.310 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: "start_epoch": 43, 2023-08-26T14:08:47.310 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: "start_stamp": "2023-08-26T14:04:58.878986+0000", 2023-08-26T14:08:47.310 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: "gid": 6118, 2023-08-26T14:08:47.311 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: "addr": "172.21.15.124:0/2216575238", 2023-08-26T14:08:47.311 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: "metadata": { 2023-08-26T14:08:47.311 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: "arch": "x86_64", 2023-08-26T14:08:47.311 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: "ceph_release": "reef", 2023-08-26T14:08:47.311 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: "ceph_version": "ceph version 18.0.0-5706-ge7eb9853 (e7eb9853fddaf661c68f207346131a7e00d8ed2f) reef (dev)", 2023-08-26T14:08:47.311 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: "ceph_version_short": "18.0.0-5706-ge7eb9853", 2023-08-26T14:08:47.312 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: "cpu": "Intel(R) Xeon(R) CPU E5-1620 v4 @ 3.50GHz", 2023-08-26T14:08:47.312 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: "distro": "rhel", 2023-08-26T14:08:47.312 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: "distro_description": "Red Hat Enterprise Linux 8.6 (Ootpa)", 2023-08-26T14:08:47.312 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: "distro_version": "8.6", 2023-08-26T14:08:47.312 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: "hostname": "smithi124", 2023-08-26T14:08:47.313 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: "id": "mirror.0", 2023-08-26T14:08:47.313 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: "instance_id": "6118", 2023-08-26T14:08:47.313 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: "kernel_description": "#1 SMP Fri Apr 15 22:12:19 EDT 2022", 2023-08-26T14:08:47.313 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: "kernel_version": "4.18.0-372.9.1.el8.x86_64", 2023-08-26T14:08:47.313 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: "mem_swap_kb": "0", 2023-08-26T14:08:47.313 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: "mem_total_kb": "32605412", 2023-08-26T14:08:47.314 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: "os": "Linux" 2023-08-26T14:08:47.314 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: }, 2023-08-26T14:08:47.314 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: "task_status": {} 2023-08-26T14:08:47.314 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: } 2023-08-26T14:08:47.314 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: } 2023-08-26T14:08:47.314 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: } 2023-08-26T14:08:47.315 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: } 2023-08-26T14:08:47.315 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:} 2023-08-26T14:08:47.315 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ CEPH_ARGS= 2023-08-26T14:08:47.316 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ ceph --cluster cluster1 service status 2023-08-26T14:08:47.710 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:{ 2023-08-26T14:08:47.710 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: "rbd-mirror": { 2023-08-26T14:08:47.710 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: "4797": { 2023-08-26T14:08:47.711 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: "status_stamp": "2023-08-26T14:08:32.898259+0000", 2023-08-26T14:08:47.711 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: "last_beacon": "2023-08-26T14:08:42.898708+0000", 2023-08-26T14:08:47.711 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: "status": { 2023-08-26T14:08:47.711 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: "json": "{\"3\":{\"name\":\"mirror\",\"callouts\":{},\"image_assigned_count\":1,\"image_error_count\":0,\"image_local_count\":0,\"image_remote_count\":1,\"image_warning_count\":0,\"instance_id\":\"4800\",\"leader\":true,\"namespaces\":{\"ns1\":{\"image_assigned_count\":0,\"image_error_count\":0,\"image_local_count\":0,\"image_remote_count\":0,\"image_warning_count\":0},\"ns2\":{\"image_assigned_count\":0,\"image_error_count\":0,\"image_local_count\":0,\"image_remote_count\":0,\"image_warning_count\":0}}},\"4\":{\"name\":\"mirror_parent\",\"callouts\":{},\"image_assigned_count\":0,\"image_error_count\":0,\"image_local_count\":0,\"image_remote_count\":0,\"image_warning_count\":0,\"instance_id\":\"4802\",\"leader\":true}}" 2023-08-26T14:08:47.711 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: } 2023-08-26T14:08:47.712 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: }, 2023-08-26T14:08:47.712 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: "5144": { 2023-08-26T14:08:47.712 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: "status_stamp": "2023-08-26T14:08:44.877983+0000", 2023-08-26T14:08:47.712 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: "last_beacon": "2023-08-26T14:08:44.877983+0000", 2023-08-26T14:08:47.713 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: "status": { 2023-08-26T14:08:47.713 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: "json": "{\"3\":{\"name\":\"mirror\",\"callouts\":{},\"image_assigned_count\":0,\"image_error_count\":0,\"image_warning_count\":0,\"instance_id\":\"5147\",\"namespaces\":{\"ns1\":{\"image_assigned_count\":0,\"image_error_count\":0,\"image_warning_count\":0},\"ns2\":{\"image_assigned_count\":0,\"image_error_count\":0,\"image_warning_count\":0}}},\"4\":{\"name\":\"mirror_parent\",\"callouts\":{},\"image_assigned_count\":0,\"image_error_count\":0,\"image_warning_count\":0,\"instance_id\":\"5150\"}}" 2023-08-26T14:08:47.713 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: } 2023-08-26T14:08:47.713 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: }, 2023-08-26T14:08:47.713 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: "5530": { 2023-08-26T14:08:47.714 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: "status_stamp": "2023-08-26T14:08:24.876984+0000", 2023-08-26T14:08:47.714 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: "last_beacon": "2023-08-26T14:08:44.877895+0000", 2023-08-26T14:08:47.714 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: "status": { 2023-08-26T14:08:47.714 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: "json": "{\"3\":{\"name\":\"mirror\",\"callouts\":{},\"image_assigned_count\":0,\"image_error_count\":0,\"image_warning_count\":0,\"instance_id\":\"5533\"},\"4\":{\"name\":\"mirror_parent\",\"callouts\":{},\"image_assigned_count\":0,\"image_error_count\":0,\"image_warning_count\":0,\"instance_id\":\"5536\"}}" 2023-08-26T14:08:47.714 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: } 2023-08-26T14:08:47.715 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: }, 2023-08-26T14:08:47.715 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: "6118": { 2023-08-26T14:08:47.715 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: "status_stamp": "2023-08-26T14:08:33.887525+0000", 2023-08-26T14:08:47.715 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: "last_beacon": "2023-08-26T14:08:43.887939+0000", 2023-08-26T14:08:47.715 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: "status": { 2023-08-26T14:08:47.716 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: "json": "{\"3\":{\"name\":\"mirror\",\"callouts\":{},\"image_assigned_count\":0,\"image_error_count\":0,\"image_warning_count\":0,\"instance_id\":\"6121\",\"namespaces\":{\"ns1\":{\"image_assigned_count\":0,\"image_error_count\":0,\"image_warning_count\":0},\"ns2\":{\"image_assigned_count\":0,\"image_error_count\":0,\"image_warning_count\":0}}},\"4\":{\"name\":\"mirror_parent\",\"callouts\":{},\"image_assigned_count\":0,\"image_error_count\":0,\"image_warning_count\":0,\"instance_id\":\"6124\"}}" 2023-08-26T14:08:47.716 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: } 2023-08-26T14:08:47.716 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: } 2023-08-26T14:08:47.716 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: } 2023-08-26T14:08:47.716 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:} 2023-08-26T14:08:47.722 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: 2023-08-26T14:08:47.723 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ echo 2023-08-26T14:08:47.723 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ for image_pool in ${POOL} ${PARENT_POOL} 2023-08-26T14:08:47.724 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ for image_ns in "" "${NS1}" "${NS2}" 2023-08-26T14:08:47.724 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:cluster1 mirror images 2023-08-26T14:08:47.724 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ echo 'cluster1 mirror images' 2023-08-26T14:08:47.725 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rbd --cluster cluster1 -p mirror --namespace '' ls -l 2023-08-26T14:08:47.759 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: 2023-08-26T14:08:47.760 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ echo 2023-08-26T14:08:47.760 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:cluster1 mirror mirror pool info 2023-08-26T14:08:47.761 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ echo 'cluster1 mirror mirror pool info' 2023-08-26T14:08:47.761 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rbd --cluster cluster1 -p mirror --namespace '' mirror pool info 2023-08-26T14:08:47.795 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:Mode: image 2023-08-26T14:08:47.795 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:Site Name: cluster1 2023-08-26T14:08:47.795 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: 2023-08-26T14:08:47.795 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:Peer Sites: 2023-08-26T14:08:47.796 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: 2023-08-26T14:08:47.796 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:UUID: 0ac69477-c0d3-495e-b791-aec031f1f9dc 2023-08-26T14:08:47.796 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:Name: cluster2 2023-08-26T14:08:47.796 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:Mirror UUID: d88cafc5-6d65-4864-9230-fe3ef515f83d 2023-08-26T14:08:47.796 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:Direction: rx-tx 2023-08-26T14:08:47.797 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:Client: client.mirror 2023-08-26T14:08:47.797 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ echo 2023-08-26T14:08:47.798 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: 2023-08-26T14:08:47.798 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:cluster1 mirror mirror pool status 2023-08-26T14:08:47.798 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ echo 'cluster1 mirror mirror pool status' 2023-08-26T14:08:47.799 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ CEPH_ARGS= 2023-08-26T14:08:47.799 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rbd --cluster cluster1 -p mirror --namespace '' mirror pool status --verbose 2023-08-26T14:08:47.833 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:health: OK 2023-08-26T14:08:47.834 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:daemon health: OK 2023-08-26T14:08:47.834 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:image health: OK 2023-08-26T14:08:47.834 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:images: 0 total 2023-08-26T14:08:47.835 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: 2023-08-26T14:08:47.835 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:DAEMONS 2023-08-26T14:08:47.835 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:service 4797: 2023-08-26T14:08:47.836 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: instance_id: 4800 2023-08-26T14:08:47.836 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: client_id: mirror.1 2023-08-26T14:08:47.836 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: hostname: smithi124 2023-08-26T14:08:47.836 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: version: 18.0.0-5706-ge7eb9853 2023-08-26T14:08:47.836 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: leader: true 2023-08-26T14:08:47.837 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: health: OK 2023-08-26T14:08:47.837 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: 2023-08-26T14:08:47.837 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:service 5144: 2023-08-26T14:08:47.837 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: instance_id: 5147 2023-08-26T14:08:47.837 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: client_id: mirror.3 2023-08-26T14:08:47.837 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: hostname: smithi124 2023-08-26T14:08:47.838 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: version: 18.0.0-5706-ge7eb9853 2023-08-26T14:08:47.838 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: leader: false 2023-08-26T14:08:47.838 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: health: OK 2023-08-26T14:08:47.838 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: 2023-08-26T14:08:47.838 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:service 5530: 2023-08-26T14:08:47.839 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: instance_id: 5533 2023-08-26T14:08:47.839 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: client_id: mirror.2 2023-08-26T14:08:47.839 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: hostname: smithi124 2023-08-26T14:08:47.839 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: version: 18.0.0-5706-ge7eb9853 2023-08-26T14:08:47.839 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: leader: false 2023-08-26T14:08:47.840 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: health: OK 2023-08-26T14:08:47.840 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: 2023-08-26T14:08:47.840 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:service 6118: 2023-08-26T14:08:47.840 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: instance_id: 6121 2023-08-26T14:08:47.840 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: client_id: mirror.0 2023-08-26T14:08:47.841 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: hostname: smithi124 2023-08-26T14:08:47.841 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: version: 18.0.0-5706-ge7eb9853 2023-08-26T14:08:47.841 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: leader: false 2023-08-26T14:08:47.841 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: health: OK 2023-08-26T14:08:47.841 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: 2023-08-26T14:08:47.842 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: 2023-08-26T14:08:47.842 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:IMAGES 2023-08-26T14:08:47.843 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ echo 2023-08-26T14:08:47.843 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ rbd --cluster cluster1 -p mirror --namespace '' ls 2023-08-26T14:08:47.875 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:cluster1 mirror rbd_mirroring omap vals 2023-08-26T14:08:47.876 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ echo 'cluster1 mirror rbd_mirroring omap vals' 2023-08-26T14:08:47.876 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rados --cluster cluster1 -p mirror --namespace '' listomapvals rbd_mirroring 2023-08-26T14:08:47.909 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:mirror_mode 2023-08-26T14:08:47.909 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:value (4 bytes) : 2023-08-26T14:08:47.909 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:00000000 01 00 00 00 |....| 2023-08-26T14:08:47.909 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:00000004 2023-08-26T14:08:47.910 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: 2023-08-26T14:08:47.910 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:mirror_peer_0ac69477-c0d3-495e-b791-aec031f1f9dc 2023-08-26T14:08:47.910 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:value (132 bytes) : 2023-08-26T14:08:47.910 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:00000000 02 01 7e 00 00 00 24 00 00 00 30 61 63 36 39 34 |..~...$...0ac694| 2023-08-26T14:08:47.910 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:00000010 37 37 2d 63 30 64 33 2d 34 39 35 65 2d 62 37 39 |77-c0d3-495e-b79| 2023-08-26T14:08:47.910 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:00000020 31 2d 61 65 63 30 33 31 66 31 66 39 64 63 08 00 |1-aec031f1f9dc..| 2023-08-26T14:08:47.911 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:00000030 00 00 63 6c 75 73 74 65 72 32 0d 00 00 00 63 6c |..cluster2....cl| 2023-08-26T14:08:47.911 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:00000040 69 65 6e 74 2e 6d 69 72 72 6f 72 ff ff ff ff ff |ient.mirror.....| 2023-08-26T14:08:47.911 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:00000050 ff ff ff 02 24 00 00 00 64 38 38 63 61 66 63 35 |....$...d88cafc5| 2023-08-26T14:08:47.911 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:00000060 2d 36 64 36 35 2d 34 38 36 34 2d 39 32 33 30 2d |-6d65-4864-9230-| 2023-08-26T14:08:47.911 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:00000070 66 65 33 65 66 35 31 35 66 38 33 64 6c 07 ea 64 |fe3ef515f83dl..d| 2023-08-26T14:08:47.912 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:00000080 cd 8d 7f 2e |....| 2023-08-26T14:08:47.912 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:00000084 2023-08-26T14:08:47.912 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: 2023-08-26T14:08:47.912 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:mirror_uuid 2023-08-26T14:08:47.912 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:value (36 bytes) : 2023-08-26T14:08:47.913 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:00000000 62 34 37 32 36 35 33 39 2d 31 61 65 66 2d 34 33 |b4726539-1aef-43| 2023-08-26T14:08:47.913 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:00000010 32 38 2d 61 62 37 62 2d 32 31 63 39 62 61 62 33 |28-ab7b-21c9bab3| 2023-08-26T14:08:47.913 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:00000020 38 63 65 64 |8ced| 2023-08-26T14:08:47.913 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:00000024 2023-08-26T14:08:47.913 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: 2023-08-26T14:08:47.914 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:cluster1 mirror rbd_mirror_leader omap vals 2023-08-26T14:08:47.914 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ echo 'cluster1 mirror rbd_mirror_leader omap vals' 2023-08-26T14:08:47.915 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rados --cluster cluster1 -p mirror --namespace '' listomapvals rbd_mirror_leader 2023-08-26T14:08:47.943 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:image_map_3ad857df-1a21-4a01-9980-6220b061adb5 2023-08-26T14:08:47.944 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:value (36 bytes) : 2023-08-26T14:08:47.944 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:00000000 01 01 1e 00 00 00 04 00 00 00 34 38 30 30 be 06 |..........4800..| 2023-08-26T14:08:47.944 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:00000010 ea 64 71 e7 c2 1a 0a 00 00 00 01 01 04 00 00 00 |.dq.............| 2023-08-26T14:08:47.944 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:00000020 00 00 00 00 |....| 2023-08-26T14:08:47.945 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:00000024 2023-08-26T14:08:47.945 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: 2023-08-26T14:08:47.945 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:instance_4800 2023-08-26T14:08:47.945 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:value (0 bytes) : 2023-08-26T14:08:47.945 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: 2023-08-26T14:08:47.946 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:instance_5147 2023-08-26T14:08:47.946 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:value (0 bytes) : 2023-08-26T14:08:47.946 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: 2023-08-26T14:08:47.946 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:instance_5533 2023-08-26T14:08:47.946 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:value (0 bytes) : 2023-08-26T14:08:47.946 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: 2023-08-26T14:08:47.947 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:instance_6121 2023-08-26T14:08:47.947 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:value (0 bytes) : 2023-08-26T14:08:47.947 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: 2023-08-26T14:08:47.948 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ echo 2023-08-26T14:08:47.948 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: 2023-08-26T14:08:47.948 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ for image_ns in "" "${NS1}" "${NS2}" 2023-08-26T14:08:47.949 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:cluster1 mirror ns1 images 2023-08-26T14:08:47.949 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ echo 'cluster1 mirror ns1 images' 2023-08-26T14:08:47.950 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rbd --cluster cluster1 -p mirror --namespace ns1 ls -l 2023-08-26T14:08:47.982 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: 2023-08-26T14:08:47.982 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:cluster1 mirrorns1 mirror pool info 2023-08-26T14:08:47.982 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ echo 2023-08-26T14:08:47.983 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ echo 'cluster1 mirrorns1 mirror pool info' 2023-08-26T14:08:47.983 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rbd --cluster cluster1 -p mirror --namespace ns1 mirror pool info 2023-08-26T14:08:48.017 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:Mode: image 2023-08-26T14:08:48.019 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ echo 2023-08-26T14:08:48.020 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: 2023-08-26T14:08:48.020 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:cluster1 mirrorns1 mirror pool status 2023-08-26T14:08:48.020 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ echo 'cluster1 mirrorns1 mirror pool status' 2023-08-26T14:08:48.020 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ CEPH_ARGS= 2023-08-26T14:08:48.021 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rbd --cluster cluster1 -p mirror --namespace ns1 mirror pool status --verbose 2023-08-26T14:08:48.056 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:health: OK 2023-08-26T14:08:48.056 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:daemon health: OK 2023-08-26T14:08:48.056 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:image health: OK 2023-08-26T14:08:48.057 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:images: 0 total 2023-08-26T14:08:48.057 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: 2023-08-26T14:08:48.057 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:DAEMONS 2023-08-26T14:08:48.057 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:service 4797: 2023-08-26T14:08:48.057 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: instance_id: 4800 2023-08-26T14:08:48.057 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: client_id: mirror.1 2023-08-26T14:08:48.058 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: hostname: smithi124 2023-08-26T14:08:48.058 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: version: 18.0.0-5706-ge7eb9853 2023-08-26T14:08:48.058 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: leader: true 2023-08-26T14:08:48.058 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: health: OK 2023-08-26T14:08:48.058 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: 2023-08-26T14:08:48.059 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:service 5144: 2023-08-26T14:08:48.059 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: instance_id: 5147 2023-08-26T14:08:48.059 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: client_id: mirror.3 2023-08-26T14:08:48.059 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: hostname: smithi124 2023-08-26T14:08:48.059 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: version: 18.0.0-5706-ge7eb9853 2023-08-26T14:08:48.060 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: leader: false 2023-08-26T14:08:48.060 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: health: OK 2023-08-26T14:08:48.060 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: 2023-08-26T14:08:48.060 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:service 5530: 2023-08-26T14:08:48.060 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: instance_id: 5533 2023-08-26T14:08:48.061 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: client_id: mirror.2 2023-08-26T14:08:48.061 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: hostname: smithi124 2023-08-26T14:08:48.061 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: version: 18.0.0-5706-ge7eb9853 2023-08-26T14:08:48.061 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: leader: false 2023-08-26T14:08:48.061 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: health: OK 2023-08-26T14:08:48.062 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: 2023-08-26T14:08:48.062 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:service 6118: 2023-08-26T14:08:48.062 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: instance_id: 6121 2023-08-26T14:08:48.062 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: client_id: mirror.0 2023-08-26T14:08:48.062 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: hostname: smithi124 2023-08-26T14:08:48.062 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: version: 18.0.0-5706-ge7eb9853 2023-08-26T14:08:48.063 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: leader: false 2023-08-26T14:08:48.063 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: health: OK 2023-08-26T14:08:48.063 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: 2023-08-26T14:08:48.063 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: 2023-08-26T14:08:48.064 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ echo 2023-08-26T14:08:48.064 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:IMAGES 2023-08-26T14:08:48.065 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ rbd --cluster cluster1 -p mirror --namespace ns1 ls 2023-08-26T14:08:48.097 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:cluster1 mirror ns1 rbd_mirroring omap vals 2023-08-26T14:08:48.097 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ echo 'cluster1 mirror ns1 rbd_mirroring omap vals' 2023-08-26T14:08:48.098 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rados --cluster cluster1 -p mirror --namespace ns1 listomapvals rbd_mirroring 2023-08-26T14:08:48.129 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:mirror_mode 2023-08-26T14:08:48.130 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:value (4 bytes) : 2023-08-26T14:08:48.130 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:00000000 01 00 00 00 |....| 2023-08-26T14:08:48.130 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:00000004 2023-08-26T14:08:48.130 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: 2023-08-26T14:08:48.130 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:mirror_uuid 2023-08-26T14:08:48.131 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:value (36 bytes) : 2023-08-26T14:08:48.131 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:00000000 61 65 62 35 37 35 34 36 2d 37 34 64 33 2d 34 30 |aeb57546-74d3-40| 2023-08-26T14:08:48.131 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:00000010 36 31 2d 38 37 66 31 2d 31 36 36 33 36 65 65 64 |61-87f1-16636eed| 2023-08-26T14:08:48.131 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:00000020 35 62 66 66 |5bff| 2023-08-26T14:08:48.131 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:00000024 2023-08-26T14:08:48.132 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: 2023-08-26T14:08:48.132 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:cluster1 mirror ns1 rbd_mirror_leader omap vals 2023-08-26T14:08:48.133 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ echo 'cluster1 mirror ns1 rbd_mirror_leader omap vals' 2023-08-26T14:08:48.133 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rados --cluster cluster1 -p mirror --namespace ns1 listomapvals rbd_mirror_leader 2023-08-26T14:08:48.164 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:instance_4800 2023-08-26T14:08:48.164 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:value (0 bytes) : 2023-08-26T14:08:48.164 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: 2023-08-26T14:08:48.164 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:instance_5147 2023-08-26T14:08:48.165 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:value (0 bytes) : 2023-08-26T14:08:48.165 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: 2023-08-26T14:08:48.165 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:instance_6121 2023-08-26T14:08:48.165 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:value (0 bytes) : 2023-08-26T14:08:48.165 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: 2023-08-26T14:08:48.166 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ echo 2023-08-26T14:08:48.166 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: 2023-08-26T14:08:48.167 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ for image_ns in "" "${NS1}" "${NS2}" 2023-08-26T14:08:48.167 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:cluster1 mirror ns2 images 2023-08-26T14:08:48.167 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ echo 'cluster1 mirror ns2 images' 2023-08-26T14:08:48.168 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rbd --cluster cluster1 -p mirror --namespace ns2 ls -l 2023-08-26T14:08:48.201 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: 2023-08-26T14:08:48.202 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ echo 2023-08-26T14:08:48.202 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:cluster1 mirrorns2 mirror pool info 2023-08-26T14:08:48.203 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ echo 'cluster1 mirrorns2 mirror pool info' 2023-08-26T14:08:48.203 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rbd --cluster cluster1 -p mirror --namespace ns2 mirror pool info 2023-08-26T14:08:48.237 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:Mode: image 2023-08-26T14:08:48.239 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ echo 2023-08-26T14:08:48.239 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: 2023-08-26T14:08:48.240 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:cluster1 mirrorns2 mirror pool status 2023-08-26T14:08:48.240 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ echo 'cluster1 mirrorns2 mirror pool status' 2023-08-26T14:08:48.240 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ CEPH_ARGS= 2023-08-26T14:08:48.240 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rbd --cluster cluster1 -p mirror --namespace ns2 mirror pool status --verbose 2023-08-26T14:08:48.277 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:health: OK 2023-08-26T14:08:48.277 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:daemon health: OK 2023-08-26T14:08:48.277 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:image health: OK 2023-08-26T14:08:48.277 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:images: 0 total 2023-08-26T14:08:48.278 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: 2023-08-26T14:08:48.278 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:DAEMONS 2023-08-26T14:08:48.278 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:service 4797: 2023-08-26T14:08:48.278 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: instance_id: 4800 2023-08-26T14:08:48.278 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: client_id: mirror.1 2023-08-26T14:08:48.278 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: hostname: smithi124 2023-08-26T14:08:48.279 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: version: 18.0.0-5706-ge7eb9853 2023-08-26T14:08:48.279 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: leader: true 2023-08-26T14:08:48.279 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: health: OK 2023-08-26T14:08:48.279 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: 2023-08-26T14:08:48.279 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:service 5144: 2023-08-26T14:08:48.280 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: instance_id: 5147 2023-08-26T14:08:48.280 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: client_id: mirror.3 2023-08-26T14:08:48.280 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: hostname: smithi124 2023-08-26T14:08:48.280 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: version: 18.0.0-5706-ge7eb9853 2023-08-26T14:08:48.280 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: leader: false 2023-08-26T14:08:48.281 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: health: OK 2023-08-26T14:08:48.281 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: 2023-08-26T14:08:48.281 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:service 5530: 2023-08-26T14:08:48.281 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: instance_id: 5533 2023-08-26T14:08:48.281 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: client_id: mirror.2 2023-08-26T14:08:48.282 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: hostname: smithi124 2023-08-26T14:08:48.282 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: version: 18.0.0-5706-ge7eb9853 2023-08-26T14:08:48.282 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: leader: false 2023-08-26T14:08:48.282 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: health: OK 2023-08-26T14:08:48.282 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: 2023-08-26T14:08:48.282 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:service 6118: 2023-08-26T14:08:48.283 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: instance_id: 6121 2023-08-26T14:08:48.283 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: client_id: mirror.0 2023-08-26T14:08:48.283 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: hostname: smithi124 2023-08-26T14:08:48.283 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: version: 18.0.0-5706-ge7eb9853 2023-08-26T14:08:48.283 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: leader: false 2023-08-26T14:08:48.284 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: health: OK 2023-08-26T14:08:48.284 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: 2023-08-26T14:08:48.284 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: 2023-08-26T14:08:48.285 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ echo 2023-08-26T14:08:48.285 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:IMAGES 2023-08-26T14:08:48.285 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ rbd --cluster cluster1 -p mirror --namespace ns2 ls 2023-08-26T14:08:48.317 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:cluster1 mirror ns2 rbd_mirroring omap vals 2023-08-26T14:08:48.317 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ echo 'cluster1 mirror ns2 rbd_mirroring omap vals' 2023-08-26T14:08:48.318 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rados --cluster cluster1 -p mirror --namespace ns2 listomapvals rbd_mirroring 2023-08-26T14:08:48.349 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:mirror_mode 2023-08-26T14:08:48.350 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:value (4 bytes) : 2023-08-26T14:08:48.350 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:00000000 01 00 00 00 |....| 2023-08-26T14:08:48.350 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:00000004 2023-08-26T14:08:48.350 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: 2023-08-26T14:08:48.350 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:mirror_uuid 2023-08-26T14:08:48.351 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:value (36 bytes) : 2023-08-26T14:08:48.351 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:00000000 38 30 64 33 66 64 37 37 2d 39 61 36 36 2d 34 31 |80d3fd77-9a66-41| 2023-08-26T14:08:48.351 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:00000010 61 64 2d 62 30 30 33 2d 30 61 30 35 33 30 39 37 |ad-b003-0a053097| 2023-08-26T14:08:48.351 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:00000020 36 65 64 38 |6ed8| 2023-08-26T14:08:48.352 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:00000024 2023-08-26T14:08:48.352 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: 2023-08-26T14:08:48.352 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ echo 'cluster1 mirror ns2 rbd_mirror_leader omap vals' 2023-08-26T14:08:48.353 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:cluster1 mirror ns2 rbd_mirror_leader omap vals 2023-08-26T14:08:48.353 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rados --cluster cluster1 -p mirror --namespace ns2 listomapvals rbd_mirror_leader 2023-08-26T14:08:48.385 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:instance_4800 2023-08-26T14:08:48.385 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:value (0 bytes) : 2023-08-26T14:08:48.385 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: 2023-08-26T14:08:48.385 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:instance_5147 2023-08-26T14:08:48.386 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:value (0 bytes) : 2023-08-26T14:08:48.386 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: 2023-08-26T14:08:48.386 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:instance_6121 2023-08-26T14:08:48.386 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:value (0 bytes) : 2023-08-26T14:08:48.386 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: 2023-08-26T14:08:48.387 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ echo 2023-08-26T14:08:48.387 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: 2023-08-26T14:08:48.388 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ for image_pool in ${POOL} ${PARENT_POOL} 2023-08-26T14:08:48.388 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ for image_ns in "" "${NS1}" "${NS2}" 2023-08-26T14:08:48.388 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ echo 'cluster1 mirror_parent images' 2023-08-26T14:08:48.389 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:cluster1 mirror_parent images 2023-08-26T14:08:48.389 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rbd --cluster cluster1 -p mirror_parent --namespace '' ls -l 2023-08-26T14:08:48.421 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: 2023-08-26T14:08:48.422 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ echo 2023-08-26T14:08:48.422 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:cluster1 mirror_parent mirror pool info 2023-08-26T14:08:48.423 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ echo 'cluster1 mirror_parent mirror pool info' 2023-08-26T14:08:48.423 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rbd --cluster cluster1 -p mirror_parent --namespace '' mirror pool info 2023-08-26T14:08:48.456 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:Mode: image 2023-08-26T14:08:48.457 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:Site Name: cluster1 2023-08-26T14:08:48.457 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: 2023-08-26T14:08:48.457 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:Peer Sites: 2023-08-26T14:08:48.457 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: 2023-08-26T14:08:48.458 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:UUID: 111b7cf3-fbb0-4f72-be72-feab89995cce 2023-08-26T14:08:48.458 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:Name: cluster2 2023-08-26T14:08:48.458 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:Mirror UUID: de94d966-4fd9-4b2c-a13b-7d2e7b1f4156 2023-08-26T14:08:48.458 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:Direction: rx-tx 2023-08-26T14:08:48.458 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:Client: client.mirror 2023-08-26T14:08:48.459 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ echo 2023-08-26T14:08:48.459 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: 2023-08-26T14:08:48.460 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ echo 'cluster1 mirror_parent mirror pool status' 2023-08-26T14:08:48.460 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:cluster1 mirror_parent mirror pool status 2023-08-26T14:08:48.461 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ CEPH_ARGS= 2023-08-26T14:08:48.461 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rbd --cluster cluster1 -p mirror_parent --namespace '' mirror pool status --verbose 2023-08-26T14:08:48.495 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:health: OK 2023-08-26T14:08:48.495 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:daemon health: OK 2023-08-26T14:08:48.495 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:image health: OK 2023-08-26T14:08:48.496 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:images: 0 total 2023-08-26T14:08:48.496 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: 2023-08-26T14:08:48.497 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:DAEMONS 2023-08-26T14:08:48.497 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:service 4797: 2023-08-26T14:08:48.497 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: instance_id: 4802 2023-08-26T14:08:48.497 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: client_id: mirror.1 2023-08-26T14:08:48.497 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: hostname: smithi124 2023-08-26T14:08:48.498 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: version: 18.0.0-5706-ge7eb9853 2023-08-26T14:08:48.498 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: leader: true 2023-08-26T14:08:48.498 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: health: OK 2023-08-26T14:08:48.498 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: 2023-08-26T14:08:48.498 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:service 5144: 2023-08-26T14:08:48.499 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: instance_id: 5150 2023-08-26T14:08:48.499 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: client_id: mirror.3 2023-08-26T14:08:48.499 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: hostname: smithi124 2023-08-26T14:08:48.500 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: version: 18.0.0-5706-ge7eb9853 2023-08-26T14:08:48.500 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: leader: false 2023-08-26T14:08:48.500 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: health: OK 2023-08-26T14:08:48.501 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: 2023-08-26T14:08:48.501 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:service 5530: 2023-08-26T14:08:48.501 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: instance_id: 5536 2023-08-26T14:08:48.501 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: client_id: mirror.2 2023-08-26T14:08:48.501 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: hostname: smithi124 2023-08-26T14:08:48.502 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: version: 18.0.0-5706-ge7eb9853 2023-08-26T14:08:48.502 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: leader: false 2023-08-26T14:08:48.502 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: health: OK 2023-08-26T14:08:48.502 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: 2023-08-26T14:08:48.503 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:service 6118: 2023-08-26T14:08:48.503 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: instance_id: 6124 2023-08-26T14:08:48.503 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: client_id: mirror.0 2023-08-26T14:08:48.503 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: hostname: smithi124 2023-08-26T14:08:48.504 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: version: 18.0.0-5706-ge7eb9853 2023-08-26T14:08:48.504 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: leader: false 2023-08-26T14:08:48.504 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: health: OK 2023-08-26T14:08:48.504 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: 2023-08-26T14:08:48.505 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: 2023-08-26T14:08:48.506 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:IMAGES 2023-08-26T14:08:48.507 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ echo 2023-08-26T14:08:48.508 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ rbd --cluster cluster1 -p mirror_parent --namespace '' ls 2023-08-26T14:08:48.534 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:cluster1 mirror_parent rbd_mirroring omap vals 2023-08-26T14:08:48.534 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ echo 'cluster1 mirror_parent rbd_mirroring omap vals' 2023-08-26T14:08:48.534 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rados --cluster cluster1 -p mirror_parent --namespace '' listomapvals rbd_mirroring 2023-08-26T14:08:48.566 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:mirror_mode 2023-08-26T14:08:48.566 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:value (4 bytes) : 2023-08-26T14:08:48.567 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:00000000 01 00 00 00 |....| 2023-08-26T14:08:48.567 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:00000004 2023-08-26T14:08:48.567 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: 2023-08-26T14:08:48.567 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:mirror_peer_111b7cf3-fbb0-4f72-be72-feab89995cce 2023-08-26T14:08:48.568 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:value (132 bytes) : 2023-08-26T14:08:48.568 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:00000000 02 01 7e 00 00 00 24 00 00 00 31 31 31 62 37 63 |..~...$...111b7c| 2023-08-26T14:08:48.568 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:00000010 66 33 2d 66 62 62 30 2d 34 66 37 32 2d 62 65 37 |f3-fbb0-4f72-be7| 2023-08-26T14:08:48.568 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:00000020 32 2d 66 65 61 62 38 39 39 39 35 63 63 65 08 00 |2-feab89995cce..| 2023-08-26T14:08:48.568 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:00000030 00 00 63 6c 75 73 74 65 72 32 0d 00 00 00 63 6c |..cluster2....cl| 2023-08-26T14:08:48.569 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:00000040 69 65 6e 74 2e 6d 69 72 72 6f 72 ff ff ff ff ff |ient.mirror.....| 2023-08-26T14:08:48.569 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:00000050 ff ff ff 02 24 00 00 00 64 65 39 34 64 39 36 36 |....$...de94d966| 2023-08-26T14:08:48.569 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:00000060 2d 34 66 64 39 2d 34 62 32 63 2d 61 31 33 62 2d |-4fd9-4b2c-a13b-| 2023-08-26T14:08:48.569 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:00000070 37 64 32 65 37 62 31 66 34 31 35 36 6c 07 ea 64 |7d2e7b1f4156l..d| 2023-08-26T14:08:48.569 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:00000080 e0 e8 88 31 |...1| 2023-08-26T14:08:48.570 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:00000084 2023-08-26T14:08:48.570 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: 2023-08-26T14:08:48.570 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:mirror_uuid 2023-08-26T14:08:48.570 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:value (36 bytes) : 2023-08-26T14:08:48.570 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:00000000 66 31 37 31 38 64 66 35 2d 31 63 66 38 2d 34 30 |f1718df5-1cf8-40| 2023-08-26T14:08:48.571 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:00000010 39 36 2d 38 38 65 35 2d 31 62 32 66 31 31 38 32 |96-88e5-1b2f1182| 2023-08-26T14:08:48.571 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:00000020 63 39 65 63 |c9ec| 2023-08-26T14:08:48.571 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:00000024 2023-08-26T14:08:48.571 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: 2023-08-26T14:08:48.572 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ echo 'cluster1 mirror_parent rbd_mirror_leader omap vals' 2023-08-26T14:08:48.572 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:cluster1 mirror_parent rbd_mirror_leader omap vals 2023-08-26T14:08:48.572 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rados --cluster cluster1 -p mirror_parent --namespace '' listomapvals rbd_mirror_leader 2023-08-26T14:08:48.600 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:instance_4802 2023-08-26T14:08:48.601 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:value (0 bytes) : 2023-08-26T14:08:48.601 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: 2023-08-26T14:08:48.601 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:instance_5150 2023-08-26T14:08:48.601 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:value (0 bytes) : 2023-08-26T14:08:48.601 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: 2023-08-26T14:08:48.601 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:instance_6124 2023-08-26T14:08:48.602 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:value (0 bytes) : 2023-08-26T14:08:48.602 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: 2023-08-26T14:08:48.602 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: 2023-08-26T14:08:48.603 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ echo 2023-08-26T14:08:48.603 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ for image_ns in "" "${NS1}" "${NS2}" 2023-08-26T14:08:48.604 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:cluster1 mirror_parent ns1 images 2023-08-26T14:08:48.604 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ echo 'cluster1 mirror_parent ns1 images' 2023-08-26T14:08:48.604 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rbd --cluster cluster1 -p mirror_parent --namespace ns1 ls -l 2023-08-26T14:08:48.635 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:rbd: namespace 'ns1' does not exist. 2023-08-26T14:08:48.635 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:rbd: listing images failed: (2) No such file or directory 2023-08-26T14:08:48.637 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ echo 2023-08-26T14:08:48.637 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: 2023-08-26T14:08:48.637 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ echo 'cluster1 mirror_parentns1 mirror pool info' 2023-08-26T14:08:48.638 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:cluster1 mirror_parentns1 mirror pool info 2023-08-26T14:08:48.638 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rbd --cluster cluster1 -p mirror_parent --namespace ns1 mirror pool info 2023-08-26T14:08:48.671 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:rbd: namespace 'ns1' does not exist. 2023-08-26T14:08:48.673 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ echo 2023-08-26T14:08:48.674 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: 2023-08-26T14:08:48.674 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:cluster1 mirror_parentns1 mirror pool status 2023-08-26T14:08:48.675 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ echo 'cluster1 mirror_parentns1 mirror pool status' 2023-08-26T14:08:48.675 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ CEPH_ARGS= 2023-08-26T14:08:48.675 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rbd --cluster cluster1 -p mirror_parent --namespace ns1 mirror pool status --verbose 2023-08-26T14:08:48.707 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:rbd: namespace 'ns1' does not exist. 2023-08-26T14:08:48.709 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: 2023-08-26T14:08:48.709 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ echo 2023-08-26T14:08:48.710 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ rbd --cluster cluster1 -p mirror_parent --namespace ns1 ls 2023-08-26T14:08:48.744 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ echo 'cluster1 mirror_parent ns1 rbd_mirroring omap vals' 2023-08-26T14:08:48.744 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:cluster1 mirror_parent ns1 rbd_mirroring omap vals 2023-08-26T14:08:48.745 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rados --cluster cluster1 -p mirror_parent --namespace ns1 listomapvals rbd_mirroring 2023-08-26T14:08:48.777 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:error getting omap keys mirror_parent/rbd_mirroring: (2) No such file or directory 2023-08-26T14:08:48.778 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:cluster1 mirror_parent ns1 rbd_mirror_leader omap vals 2023-08-26T14:08:48.779 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ echo 'cluster1 mirror_parent ns1 rbd_mirror_leader omap vals' 2023-08-26T14:08:48.779 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rados --cluster cluster1 -p mirror_parent --namespace ns1 listomapvals rbd_mirror_leader 2023-08-26T14:08:48.811 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:error getting omap keys mirror_parent/rbd_mirror_leader: (2) No such file or directory 2023-08-26T14:08:48.812 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ echo 2023-08-26T14:08:48.813 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: 2023-08-26T14:08:48.813 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ for image_ns in "" "${NS1}" "${NS2}" 2023-08-26T14:08:48.814 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:cluster1 mirror_parent ns2 images 2023-08-26T14:08:48.814 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ echo 'cluster1 mirror_parent ns2 images' 2023-08-26T14:08:48.814 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rbd --cluster cluster1 -p mirror_parent --namespace ns2 ls -l 2023-08-26T14:08:48.846 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:rbd: namespace 'ns2' does not exist. 2023-08-26T14:08:48.846 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:rbd: listing images failed: (2) No such file or directory 2023-08-26T14:08:48.848 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: 2023-08-26T14:08:48.848 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ echo 2023-08-26T14:08:48.849 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:cluster1 mirror_parentns2 mirror pool info 2023-08-26T14:08:48.849 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ echo 'cluster1 mirror_parentns2 mirror pool info' 2023-08-26T14:08:48.849 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rbd --cluster cluster1 -p mirror_parent --namespace ns2 mirror pool info 2023-08-26T14:08:48.882 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:rbd: namespace 'ns2' does not exist. 2023-08-26T14:08:48.887 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ echo 2023-08-26T14:08:48.888 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: 2023-08-26T14:08:48.888 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:cluster1 mirror_parentns2 mirror pool status 2023-08-26T14:08:48.889 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ echo 'cluster1 mirror_parentns2 mirror pool status' 2023-08-26T14:08:48.889 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ CEPH_ARGS= 2023-08-26T14:08:48.889 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rbd --cluster cluster1 -p mirror_parent --namespace ns2 mirror pool status --verbose 2023-08-26T14:08:48.918 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:rbd: namespace 'ns2' does not exist. 2023-08-26T14:08:48.920 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: 2023-08-26T14:08:48.920 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ echo 2023-08-26T14:08:48.921 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ rbd --cluster cluster1 -p mirror_parent --namespace ns2 ls 2023-08-26T14:08:48.955 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:cluster1 mirror_parent ns2 rbd_mirroring omap vals 2023-08-26T14:08:48.956 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ echo 'cluster1 mirror_parent ns2 rbd_mirroring omap vals' 2023-08-26T14:08:48.956 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rados --cluster cluster1 -p mirror_parent --namespace ns2 listomapvals rbd_mirroring 2023-08-26T14:08:48.988 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:error getting omap keys mirror_parent/rbd_mirroring: (2) No such file or directory 2023-08-26T14:08:48.989 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ echo 'cluster1 mirror_parent ns2 rbd_mirror_leader omap vals' 2023-08-26T14:08:48.990 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:cluster1 mirror_parent ns2 rbd_mirror_leader omap vals 2023-08-26T14:08:48.990 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rados --cluster cluster1 -p mirror_parent --namespace ns2 listomapvals rbd_mirror_leader 2023-08-26T14:08:49.022 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:error getting omap keys mirror_parent/rbd_mirror_leader: (2) No such file or directory 2023-08-26T14:08:49.024 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ echo 2023-08-26T14:08:49.024 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: 2023-08-26T14:08:49.025 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ for cluster in ${CLUSTER1} ${CLUSTER2} 2023-08-26T14:08:49.025 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ echo 'cluster2 status' 2023-08-26T14:08:49.025 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:cluster2 status 2023-08-26T14:08:49.026 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ CEPH_ARGS= 2023-08-26T14:08:49.026 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ ceph --cluster cluster2 -s 2023-08-26T14:13:49.115 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:2023-08-26T14:13:49.113+0000 7f0057ab7700 0 monclient(hunting): authenticate timed out after 300 2023-08-26T14:13:49.115 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:[errno 110] RADOS timed out (error connecting to the cluster) 2023-08-26T14:13:49.117 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ CEPH_ARGS= 2023-08-26T14:13:49.117 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ ceph --cluster cluster2 service dump 2023-08-26T14:17:09.714 INFO:tasks.rbd_mirror_thrash:Failed to stop cluster2.client.mirror.0 2023-08-26T14:17:09.715 INFO:tasks.rbd_mirror.cluster2.client.mirror.0:Sent signal 6 2023-08-26T14:17:09.715 ERROR:tasks.rbd_mirror_thrash:exception: Traceback (most recent call last): File "/home/teuthworker/src/github.com_ceph_ceph-c_e7eb9853fddaf661c68f207346131a7e00d8ed2f/qa/tasks/rbd_mirror_thrash.py", line 84, in _run self.do_thrash() File "/home/teuthworker/src/github.com_ceph_ceph-c_e7eb9853fddaf661c68f207346131a7e00d8ed2f/qa/tasks/rbd_mirror_thrash.py", line 156, in do_thrash run.wait([daemon.proc], timeout=600) File "/home/teuthworker/src/git.ceph.com_teuthology_449a1bc2027504e7b3c3d7b30fa4178906581da7/teuthology/orchestra/run.py", line 473, in wait check_time() File "/home/teuthworker/src/git.ceph.com_teuthology_449a1bc2027504e7b3c3d7b30fa4178906581da7/teuthology/contextutil.py", line 134, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (101) after waiting for 600 seconds 2023-08-26T14:17:09.716 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi124.stderr:*** Caught signal (Aborted) ** 2023-08-26T14:17:09.716 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi124.stderr: in thread 7fc0ce19a8c0 thread_name:rbd-mirror 2023-08-26T14:17:09.717 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi124.stderr: ceph version 18.0.0-5706-ge7eb9853 (e7eb9853fddaf661c68f207346131a7e00d8ed2f) reef (dev) 2023-08-26T14:17:09.717 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi124.stderr: 1: /lib64/libpthread.so.0(+0x12ce0) [0x7fc0d4278ce0] 2023-08-26T14:17:09.717 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi124.stderr: 2: pthread_cond_wait() 2023-08-26T14:17:09.717 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi124.stderr: 3: (std::condition_variable::wait(std::unique_lock&)+0x10) [0x7fc0d292f8f0] 2023-08-26T14:17:09.717 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi124.stderr: 4: /lib64/librados.so.2(+0xb3a8b) [0x7fc0d623fa8b] 2023-08-26T14:17:09.718 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi124.stderr: 5: (librados::v14_2_0::IoCtx::operate(std::__cxx11::basic_string, std::allocator > const&, librados::v14_2_0::ObjectReadOperation*, ceph::buffer::v15_2_0::list*)+0x4d) [0x7fc0d621074d] 2023-08-26T14:17:09.718 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi124.stderr: 6: (librbd::cls_client::mirror_mode_get(librados::v14_2_0::IoCtx*, cls::rbd::MirrorMode*)+0x96) [0x5567dd7d8d66] 2023-08-26T14:17:09.718 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi124.stderr: 7: (rbd::mirror::ClusterWatcher::read_pool_peers(std::map >, std::less, std::allocator > > > >*)+0x694) [0x5567dd046374] 2023-08-26T14:17:09.718 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi124.stderr: 8: (rbd::mirror::ClusterWatcher::refresh_pools()+0x7c) [0x5567dd048a9c] 2023-08-26T14:17:09.718 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi124.stderr: 9: (rbd::mirror::Mirror::run()+0xb4) [0x5567dd007294] 2023-08-26T14:17:09.719 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi124.stderr: 10: main() 2023-08-26T14:17:09.719 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi124.stderr: 11: __libc_start_main() 2023-08-26T14:17:09.719 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi124.stderr: 12: _start() 2023-08-26T14:17:09.719 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi124.stderr:2023-08-26T14:17:09.716+0000 7fc0ce19a8c0 -1 *** Caught signal (Aborted) ** 2023-08-26T14:17:09.719 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi124.stderr: in thread 7fc0ce19a8c0 thread_name:rbd-mirror 2023-08-26T14:17:09.720 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi124.stderr: 2023-08-26T14:17:09.720 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi124.stderr: ceph version 18.0.0-5706-ge7eb9853 (e7eb9853fddaf661c68f207346131a7e00d8ed2f) reef (dev) 2023-08-26T14:17:09.720 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi124.stderr: 1: /lib64/libpthread.so.0(+0x12ce0) [0x7fc0d4278ce0] 2023-08-26T14:17:09.720 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi124.stderr: 2: pthread_cond_wait() 2023-08-26T14:17:09.720 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi124.stderr: 3: (std::condition_variable::wait(std::unique_lock&)+0x10) [0x7fc0d292f8f0] 2023-08-26T14:17:09.721 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi124.stderr: 4: /lib64/librados.so.2(+0xb3a8b) [0x7fc0d623fa8b] 2023-08-26T14:17:09.721 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi124.stderr: 5: (librados::v14_2_0::IoCtx::operate(std::__cxx11::basic_string, std::allocator > const&, librados::v14_2_0::ObjectReadOperation*, ceph::buffer::v15_2_0::list*)+0x4d) [0x7fc0d621074d] 2023-08-26T14:17:09.721 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi124.stderr: 6: (librbd::cls_client::mirror_mode_get(librados::v14_2_0::IoCtx*, cls::rbd::MirrorMode*)+0x96) [0x5567dd7d8d66] 2023-08-26T14:17:09.721 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi124.stderr: 7: (rbd::mirror::ClusterWatcher::read_pool_peers(std::map >, std::less, std::allocator > > > >*)+0x694) [0x5567dd046374] 2023-08-26T14:17:09.721 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi124.stderr: 8: (rbd::mirror::ClusterWatcher::refresh_pools()+0x7c) [0x5567dd048a9c] 2023-08-26T14:17:09.722 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi124.stderr: 9: (rbd::mirror::Mirror::run()+0xb4) [0x5567dd007294] 2023-08-26T14:17:09.722 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi124.stderr: 10: main() 2023-08-26T14:17:09.722 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi124.stderr: 11: __libc_start_main() 2023-08-26T14:17:09.722 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi124.stderr: 12: _start() 2023-08-26T14:17:09.722 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi124.stderr: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2023-08-26T14:17:09.723 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi124.stderr: 2023-08-26T14:17:09.723 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi124.stderr: -8277> 2023-08-26T13:59:45.686+0000 7fc0c3135700 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x5567e0d511e0 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2023-08-26T14:17:09.724 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi124.stderr: -8274> 2023-08-26T13:59:45.686+0000 7fc0c9942700 -1 rbd::mirror::ImageReplayer: 0x5567e030db80 [3/072383ad-81e1-4717-a595-674819d66895] operator(): start failed: (85) Interrupted system call should be restarted 2023-08-26T14:17:09.724 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi124.stderr: -8142> 2023-08-26T13:59:53.516+0000 7fc0c3936700 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x5567e0b604e0 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2023-08-26T14:17:09.724 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi124.stderr: -7961> 2023-08-26T14:00:11.424+0000 7fc0c3135700 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x5567e0b60000 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2023-08-26T14:17:09.724 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi124.stderr: -7958> 2023-08-26T14:00:11.424+0000 7fc0c9942700 -1 rbd::mirror::ImageReplayer: 0x5567e030ca00 [3/57b5e313-8920-4ded-8ea4-ab852f9e42c2] operator(): start failed: (85) Interrupted system call should be restarted 2023-08-26T14:17:09.725 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi124.stderr: -6815> 2023-08-26T14:00:23.530+0000 7fc0c3936700 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x5567e0b61a00 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2023-08-26T14:17:09.725 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi124.stderr: -6812> 2023-08-26T14:00:23.530+0000 7fc0c9942700 -1 rbd::mirror::ImageReplayer: 0x5567e0ba2280 [3/04903506-ddd2-48b7-835f-3797d11e35d3] operator(): start failed: (85) Interrupted system call should be restarted 2023-08-26T14:17:09.725 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi124.stderr: -6679> 2023-08-26T14:00:31.518+0000 7fc0c3936700 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x5567e0d29ba0 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2023-08-26T14:17:09.726 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi124.stderr: -6676> 2023-08-26T14:00:31.518+0000 7fc0ca143700 -1 rbd::mirror::ImageReplayer: 0x5567e030d400 [3/ae7a1e38-d406-47df-97d4-9cbf52592bb7] operator(): start failed: (85) Interrupted system call should be restarted 2023-08-26T14:17:09.726 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi124.stderr: -6585> 2023-08-26T14:00:33.520+0000 7fc0c3936700 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x5567e0b60820 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2023-08-26T14:17:09.726 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi124.stderr: -6582> 2023-08-26T14:00:33.520+0000 7fc0c9942700 -1 rbd::mirror::ImageReplayer: 0x5567e0ba2000 [3/ef7edecb-ca31-4518-87d6-58c3dfe520f7] operator(): start failed: (85) Interrupted system call should be restarted 2023-08-26T14:17:09.727 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi124.stderr: -5696> 2023-08-26T14:00:51.437+0000 7fc0c3936700 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x5567e0dda4e0 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2023-08-26T14:17:09.727 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi124.stderr: -5693> 2023-08-26T14:00:51.437+0000 7fc0ca143700 -1 rbd::mirror::ImageReplayer: 0x5567e030cf00 [3/92367d6b-5947-4538-a25d-c7bdd622153a] operator(): start failed: (85) Interrupted system call should be restarted 2023-08-26T14:17:09.727 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi124.stderr: -5602> 2023-08-26T14:00:53.479+0000 7fc0c3936700 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x5567e0d48340 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2023-08-26T14:17:09.727 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi124.stderr: -5599> 2023-08-26T14:00:53.479+0000 7fc0c9942700 -1 rbd::mirror::ImageReplayer: 0x5567e030d180 [3/8068d21b-b929-4466-9d32-bbd96d0ca61c] operator(): start failed: (85) Interrupted system call should be restarted 2023-08-26T14:17:09.728 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi124.stderr: -5486> 2023-08-26T14:00:59.409+0000 7fc0c3135700 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x5567e0d48000 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2023-08-26T14:17:09.728 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi124.stderr: -5483> 2023-08-26T14:00:59.409+0000 7fc0c9942700 -1 rbd::mirror::ImageReplayer: 0x5567e030d680 [3/e777bb16-afba-4c52-994a-6980c13f1760] operator(): start failed: (85) Interrupted system call should be restarted 2023-08-26T14:17:09.728 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi124.stderr: -5366> 2023-08-26T14:01:03.463+0000 7fc0c3135700 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x5567e0b604e0 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2023-08-26T14:17:09.728 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi124.stderr: -5363> 2023-08-26T14:01:03.463+0000 7fc0c9942700 -1 rbd::mirror::ImageReplayer: 0x5567e030d900 [3/bfd70442-dd63-468e-9119-77e3357316c5] operator(): start failed: (85) Interrupted system call should be restarted 2023-08-26T14:17:09.728 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi124.stderr: -4896> 2023-08-26T14:01:17.541+0000 7fc0c3936700 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x5567e0dda680 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2023-08-26T14:17:09.729 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi124.stderr: -4893> 2023-08-26T14:01:17.541+0000 7fc0ca143700 -1 rbd::mirror::ImageReplayer: 0x5567e030cc80 [3/6721bd5f-42ff-4eab-84c6-4ca33fff75c0] operator(): start failed: (85) Interrupted system call should be restarted 2023-08-26T14:17:09.729 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi124.stderr: -4803> 2023-08-26T14:01:19.448+0000 7fc0c3936700 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x5567e0dda820 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2023-08-26T14:17:09.729 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi124.stderr: -4800> 2023-08-26T14:01:19.448+0000 7fc0ca143700 -1 rbd::mirror::ImageReplayer: 0x5567e0ba2780 [3/5f1ee9ae-1ac1-4156-b437-c09951a7762e] operator(): start failed: (85) Interrupted system call should be restarted 2023-08-26T14:17:09.730 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi124.stderr: -1791> 2023-08-26T14:07:02.568+0000 7fc0ca944700 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.0 (PID: 131554) UID: 1000 2023-08-26T14:17:09.731 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi124.stderr: 0> 2023-08-26T14:17:09.716+0000 7fc0ce19a8c0 -1 *** Caught signal (Aborted) ** 2023-08-26T14:17:09.731 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi124.stderr: in thread 7fc0ce19a8c0 thread_name:rbd-mirror 2023-08-26T14:17:09.731 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi124.stderr: 2023-08-26T14:17:09.731 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi124.stderr: ceph version 18.0.0-5706-ge7eb9853 (e7eb9853fddaf661c68f207346131a7e00d8ed2f) reef (dev) 2023-08-26T14:17:09.732 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi124.stderr: 1: /lib64/libpthread.so.0(+0x12ce0) [0x7fc0d4278ce0] 2023-08-26T14:17:09.732 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi124.stderr: 2: pthread_cond_wait() 2023-08-26T14:17:09.732 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi124.stderr: 3: (std::condition_variable::wait(std::unique_lock&)+0x10) [0x7fc0d292f8f0] 2023-08-26T14:17:09.732 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi124.stderr: 4: /lib64/librados.so.2(+0xb3a8b) [0x7fc0d623fa8b] 2023-08-26T14:17:09.732 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi124.stderr: 5: (librados::v14_2_0::IoCtx::operate(std::__cxx11::basic_string, std::allocator > const&, librados::v14_2_0::ObjectReadOperation*, ceph::buffer::v15_2_0::list*)+0x4d) [0x7fc0d621074d] 2023-08-26T14:17:09.733 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi124.stderr: 6: (librbd::cls_client::mirror_mode_get(librados::v14_2_0::IoCtx*, cls::rbd::MirrorMode*)+0x96) [0x5567dd7d8d66] 2023-08-26T14:17:09.733 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi124.stderr: 7: (rbd::mirror::ClusterWatcher::read_pool_peers(std::map >, std::less, std::allocator > > > >*)+0x694) [0x5567dd046374] 2023-08-26T14:17:09.733 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi124.stderr: 8: (rbd::mirror::ClusterWatcher::refresh_pools()+0x7c) [0x5567dd048a9c] 2023-08-26T14:17:09.733 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi124.stderr: 9: (rbd::mirror::Mirror::run()+0xb4) [0x5567dd007294] 2023-08-26T14:17:09.733 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi124.stderr: 10: main() 2023-08-26T14:17:09.734 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi124.stderr: 11: __libc_start_main() 2023-08-26T14:17:09.734 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi124.stderr: 12: _start() 2023-08-26T14:17:09.734 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi124.stderr: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2023-08-26T14:17:09.734 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi124.stderr: 2023-08-26T14:17:09.736 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi124.stderr: -9999> 2023-08-26T13:59:45.686+0000 7fc0c3135700 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x5567e0d511e0 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2023-08-26T14:17:09.736 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi124.stderr: -9998> 2023-08-26T13:59:45.686+0000 7fc0c9942700 -1 rbd::mirror::ImageReplayer: 0x5567e030db80 [3/072383ad-81e1-4717-a595-674819d66895] operator(): start failed: (85) Interrupted system call should be restarted 2023-08-26T14:17:09.736 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi124.stderr: -9997> 2023-08-26T13:59:53.516+0000 7fc0c3936700 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x5567e0b604e0 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2023-08-26T14:17:09.736 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi124.stderr: -9996> 2023-08-26T14:00:11.424+0000 7fc0c3135700 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x5567e0b60000 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2023-08-26T14:17:09.737 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi124.stderr: -9995> 2023-08-26T14:00:11.424+0000 7fc0c9942700 -1 rbd::mirror::ImageReplayer: 0x5567e030ca00 [3/57b5e313-8920-4ded-8ea4-ab852f9e42c2] operator(): start failed: (85) Interrupted system call should be restarted 2023-08-26T14:17:09.737 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi124.stderr: -9994> 2023-08-26T14:00:23.530+0000 7fc0c3936700 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x5567e0b61a00 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2023-08-26T14:17:09.737 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi124.stderr: -9993> 2023-08-26T14:00:23.530+0000 7fc0c9942700 -1 rbd::mirror::ImageReplayer: 0x5567e0ba2280 [3/04903506-ddd2-48b7-835f-3797d11e35d3] operator(): start failed: (85) Interrupted system call should be restarted 2023-08-26T14:17:09.737 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi124.stderr: -9992> 2023-08-26T14:00:31.518+0000 7fc0c3936700 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x5567e0d29ba0 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2023-08-26T14:17:09.737 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi124.stderr: -9991> 2023-08-26T14:00:31.518+0000 7fc0ca143700 -1 rbd::mirror::ImageReplayer: 0x5567e030d400 [3/ae7a1e38-d406-47df-97d4-9cbf52592bb7] operator(): start failed: (85) Interrupted system call should be restarted 2023-08-26T14:17:09.737 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi124.stderr: -9990> 2023-08-26T14:00:33.520+0000 7fc0c3936700 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x5567e0b60820 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2023-08-26T14:17:09.738 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi124.stderr: -9989> 2023-08-26T14:00:33.520+0000 7fc0c9942700 -1 rbd::mirror::ImageReplayer: 0x5567e0ba2000 [3/ef7edecb-ca31-4518-87d6-58c3dfe520f7] operator(): start failed: (85) Interrupted system call should be restarted 2023-08-26T14:17:09.738 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi124.stderr: -9988> 2023-08-26T14:00:51.437+0000 7fc0c3936700 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x5567e0dda4e0 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2023-08-26T14:17:09.738 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi124.stderr: -9987> 2023-08-26T14:00:51.437+0000 7fc0ca143700 -1 rbd::mirror::ImageReplayer: 0x5567e030cf00 [3/92367d6b-5947-4538-a25d-c7bdd622153a] operator(): start failed: (85) Interrupted system call should be restarted 2023-08-26T14:17:09.738 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi124.stderr: -9986> 2023-08-26T14:00:53.479+0000 7fc0c3936700 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x5567e0d48340 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2023-08-26T14:17:09.738 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi124.stderr: -9985> 2023-08-26T14:00:53.479+0000 7fc0c9942700 -1 rbd::mirror::ImageReplayer: 0x5567e030d180 [3/8068d21b-b929-4466-9d32-bbd96d0ca61c] operator(): start failed: (85) Interrupted system call should be restarted 2023-08-26T14:17:09.739 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi124.stderr: -9984> 2023-08-26T14:00:59.409+0000 7fc0c3135700 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x5567e0d48000 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2023-08-26T14:17:09.739 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi124.stderr: -9983> 2023-08-26T14:00:59.409+0000 7fc0c9942700 -1 rbd::mirror::ImageReplayer: 0x5567e030d680 [3/e777bb16-afba-4c52-994a-6980c13f1760] operator(): start failed: (85) Interrupted system call should be restarted 2023-08-26T14:17:09.739 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi124.stderr: -9982> 2023-08-26T14:01:03.463+0000 7fc0c3135700 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x5567e0b604e0 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2023-08-26T14:17:09.739 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi124.stderr: -9981> 2023-08-26T14:01:03.463+0000 7fc0c9942700 -1 rbd::mirror::ImageReplayer: 0x5567e030d900 [3/bfd70442-dd63-468e-9119-77e3357316c5] operator(): start failed: (85) Interrupted system call should be restarted 2023-08-26T14:17:09.739 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi124.stderr: -9980> 2023-08-26T14:01:17.541+0000 7fc0c3936700 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x5567e0dda680 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2023-08-26T14:17:09.740 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi124.stderr: -9979> 2023-08-26T14:01:17.541+0000 7fc0ca143700 -1 rbd::mirror::ImageReplayer: 0x5567e030cc80 [3/6721bd5f-42ff-4eab-84c6-4ca33fff75c0] operator(): start failed: (85) Interrupted system call should be restarted 2023-08-26T14:17:09.740 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi124.stderr: -9978> 2023-08-26T14:01:19.448+0000 7fc0c3936700 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x5567e0dda820 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2023-08-26T14:17:09.740 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi124.stderr: -9977> 2023-08-26T14:01:19.448+0000 7fc0ca143700 -1 rbd::mirror::ImageReplayer: 0x5567e0ba2780 [3/5f1ee9ae-1ac1-4156-b437-c09951a7762e] operator(): start failed: (85) Interrupted system call should be restarted 2023-08-26T14:17:09.740 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi124.stderr: -9976> 2023-08-26T14:07:02.568+0000 7fc0ca944700 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.0 (PID: 131554) UID: 1000 2023-08-26T14:17:09.740 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi124.stderr: -9975> 2023-08-26T14:17:09.716+0000 7fc0ce19a8c0 -1 *** Caught signal (Aborted) ** 2023-08-26T14:17:09.741 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi124.stderr: in thread 7fc0ce19a8c0 thread_name:rbd-mirror 2023-08-26T14:17:09.741 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi124.stderr: 2023-08-26T14:17:09.741 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi124.stderr: ceph version 18.0.0-5706-ge7eb9853 (e7eb9853fddaf661c68f207346131a7e00d8ed2f) reef (dev) 2023-08-26T14:17:09.741 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi124.stderr: 1: /lib64/libpthread.so.0(+0x12ce0) [0x7fc0d4278ce0] 2023-08-26T14:17:09.741 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi124.stderr: 2: pthread_cond_wait() 2023-08-26T14:17:09.742 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi124.stderr: 3: (std::condition_variable::wait(std::unique_lock&)+0x10) [0x7fc0d292f8f0] 2023-08-26T14:17:09.742 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi124.stderr: 4: /lib64/librados.so.2(+0xb3a8b) [0x7fc0d623fa8b] 2023-08-26T14:17:09.742 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi124.stderr: 5: (librados::v14_2_0::IoCtx::operate(std::__cxx11::basic_string, std::allocator > const&, librados::v14_2_0::ObjectReadOperation*, ceph::buffer::v15_2_0::list*)+0x4d) [0x7fc0d621074d] 2023-08-26T14:17:09.742 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi124.stderr: 6: (librbd::cls_client::mirror_mode_get(librados::v14_2_0::IoCtx*, cls::rbd::MirrorMode*)+0x96) [0x5567dd7d8d66] 2023-08-26T14:17:09.742 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi124.stderr: 7: (rbd::mirror::ClusterWatcher::read_pool_peers(std::map >, std::less, std::allocator > > > >*)+0x694) [0x5567dd046374] 2023-08-26T14:17:09.743 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi124.stderr: 8: (rbd::mirror::ClusterWatcher::refresh_pools()+0x7c) [0x5567dd048a9c] 2023-08-26T14:17:09.743 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi124.stderr: 9: (rbd::mirror::Mirror::run()+0xb4) [0x5567dd007294] 2023-08-26T14:17:09.743 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi124.stderr: 10: main() 2023-08-26T14:17:09.743 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi124.stderr: 11: __libc_start_main() 2023-08-26T14:17:09.743 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi124.stderr: 12: _start() 2023-08-26T14:17:09.743 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi124.stderr: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2023-08-26T14:17:09.744 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi124.stderr: 2023-08-26T14:17:09.923 INFO:tasks.rbd_mirror_thrash:Failed to stop cluster1.client.mirror.2 2023-08-26T14:17:09.924 INFO:tasks.rbd_mirror.cluster1.client.mirror.2:Sent signal 6 2023-08-26T14:17:09.925 ERROR:tasks.rbd_mirror_thrash:exception: Traceback (most recent call last): File "/home/teuthworker/src/github.com_ceph_ceph-c_e7eb9853fddaf661c68f207346131a7e00d8ed2f/qa/tasks/rbd_mirror_thrash.py", line 84, in _run self.do_thrash() File "/home/teuthworker/src/github.com_ceph_ceph-c_e7eb9853fddaf661c68f207346131a7e00d8ed2f/qa/tasks/rbd_mirror_thrash.py", line 156, in do_thrash run.wait([daemon.proc], timeout=600) File "/home/teuthworker/src/git.ceph.com_teuthology_449a1bc2027504e7b3c3d7b30fa4178906581da7/teuthology/orchestra/run.py", line 473, in wait check_time() File "/home/teuthworker/src/git.ceph.com_teuthology_449a1bc2027504e7b3c3d7b30fa4178906581da7/teuthology/contextutil.py", line 134, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (101) after waiting for 600 seconds 2023-08-26T14:17:09.926 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi124.stderr:*** Caught signal (Aborted) ** 2023-08-26T14:17:09.926 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi124.stderr: in thread 7f733d1d58c0 thread_name:rbd-mirror 2023-08-26T14:17:09.927 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi124.stderr: ceph version 18.0.0-5706-ge7eb9853 (e7eb9853fddaf661c68f207346131a7e00d8ed2f) reef (dev) 2023-08-26T14:17:09.927 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi124.stderr: 1: /lib64/libpthread.so.0(+0x12ce0) [0x7f73432b3ce0] 2023-08-26T14:17:09.928 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi124.stderr: 2: /lib64/libpthread.so.0(+0x96bd) [0x7f73432aa6bd] 2023-08-26T14:17:09.928 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi124.stderr: 3: (Thread::join(void**)+0x33) [0x7f7343f934e3] 2023-08-26T14:17:09.928 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi124.stderr: 4: (rbd::mirror::PoolReplayer::shut_down()+0x46a) [0x55cf9c6fb3aa] 2023-08-26T14:17:09.929 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi124.stderr: 5: (rbd::mirror::PoolReplayer::~PoolReplayer()+0x10) [0x55cf9c6fb510] 2023-08-26T14:17:09.929 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi124.stderr: 6: rbd-mirror(+0x55d2e4) [0x55cf9c6e92e4] 2023-08-26T14:17:09.929 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi124.stderr: 7: (rbd::mirror::Mirror::~Mirror()+0x109) [0x55cf9c6e9dc9] 2023-08-26T14:17:09.930 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi124.stderr: 8: main() 2023-08-26T14:17:09.930 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi124.stderr: 9: __libc_start_main() 2023-08-26T14:17:09.930 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi124.stderr: 10: _start() 2023-08-26T14:17:09.931 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi124.stderr:2023-08-26T14:17:09.925+0000 7f733d1d58c0 -1 *** Caught signal (Aborted) ** 2023-08-26T14:17:09.931 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi124.stderr: in thread 7f733d1d58c0 thread_name:rbd-mirror 2023-08-26T14:17:09.931 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi124.stderr: 2023-08-26T14:17:09.932 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi124.stderr: ceph version 18.0.0-5706-ge7eb9853 (e7eb9853fddaf661c68f207346131a7e00d8ed2f) reef (dev) 2023-08-26T14:17:09.932 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi124.stderr: 1: /lib64/libpthread.so.0(+0x12ce0) [0x7f73432b3ce0] 2023-08-26T14:17:09.932 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi124.stderr: 2: /lib64/libpthread.so.0(+0x96bd) [0x7f73432aa6bd] 2023-08-26T14:17:09.933 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi124.stderr: 3: (Thread::join(void**)+0x33) [0x7f7343f934e3] 2023-08-26T14:17:09.933 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi124.stderr: 4: (rbd::mirror::PoolReplayer::shut_down()+0x46a) [0x55cf9c6fb3aa] 2023-08-26T14:17:09.933 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi124.stderr: 5: (rbd::mirror::PoolReplayer::~PoolReplayer()+0x10) [0x55cf9c6fb510] 2023-08-26T14:17:09.934 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi124.stderr: 6: rbd-mirror(+0x55d2e4) [0x55cf9c6e92e4] 2023-08-26T14:17:09.934 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi124.stderr: 7: (rbd::mirror::Mirror::~Mirror()+0x109) [0x55cf9c6e9dc9] 2023-08-26T14:17:09.934 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi124.stderr: 8: main() 2023-08-26T14:17:09.935 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi124.stderr: 9: __libc_start_main() 2023-08-26T14:17:09.935 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi124.stderr: 10: _start() 2023-08-26T14:17:09.935 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi124.stderr: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2023-08-26T14:17:09.936 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi124.stderr: 2023-08-26T14:17:09.938 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi124.stderr: -2638> 2023-08-26T14:06:57.785+0000 7f733997f700 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.2 (PID: 130412) UID: 1000 2023-08-26T14:17:09.940 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi124.stderr: 0> 2023-08-26T14:17:09.925+0000 7f733d1d58c0 -1 *** Caught signal (Aborted) ** 2023-08-26T14:17:09.940 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi124.stderr: in thread 7f733d1d58c0 thread_name:rbd-mirror 2023-08-26T14:17:09.941 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi124.stderr: 2023-08-26T14:17:09.941 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi124.stderr: ceph version 18.0.0-5706-ge7eb9853 (e7eb9853fddaf661c68f207346131a7e00d8ed2f) reef (dev) 2023-08-26T14:17:09.941 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi124.stderr: 1: /lib64/libpthread.so.0(+0x12ce0) [0x7f73432b3ce0] 2023-08-26T14:17:09.941 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi124.stderr: 2: /lib64/libpthread.so.0(+0x96bd) [0x7f73432aa6bd] 2023-08-26T14:17:09.941 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi124.stderr: 3: (Thread::join(void**)+0x33) [0x7f7343f934e3] 2023-08-26T14:17:09.941 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi124.stderr: 4: (rbd::mirror::PoolReplayer::shut_down()+0x46a) [0x55cf9c6fb3aa] 2023-08-26T14:17:09.942 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi124.stderr: 5: (rbd::mirror::PoolReplayer::~PoolReplayer()+0x10) [0x55cf9c6fb510] 2023-08-26T14:17:09.942 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi124.stderr: 6: rbd-mirror(+0x55d2e4) [0x55cf9c6e92e4] 2023-08-26T14:17:09.942 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi124.stderr: 7: (rbd::mirror::Mirror::~Mirror()+0x109) [0x55cf9c6e9dc9] 2023-08-26T14:17:09.942 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi124.stderr: 8: main() 2023-08-26T14:17:09.942 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi124.stderr: 9: __libc_start_main() 2023-08-26T14:17:09.943 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi124.stderr: 10: _start() 2023-08-26T14:17:09.943 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi124.stderr: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2023-08-26T14:17:09.943 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi124.stderr: 2023-08-26T14:17:10.116 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi124.stderr:daemon-helper: command crashed with signal 6 2023-08-26T14:17:10.926 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi124.stderr:daemon-helper: command crashed with signal 11 2023-08-26T14:17:14.104 INFO:tasks.daemonwatchdog.daemon_watchdog:thrasher.rbd_mirror.[cluster1] failed 2023-08-26T14:17:14.104 INFO:tasks.daemonwatchdog.daemon_watchdog:BARK! unmounting mounts and killing all daemons 2023-08-26T14:17:14.607 INFO:tasks.ceph.cluster1.osd.0:Sent signal 15 2023-08-26T14:17:14.608 INFO:tasks.ceph.cluster1.osd.1:Sent signal 15 2023-08-26T14:17:14.608 INFO:tasks.ceph.cluster1.osd.2:Sent signal 15 2023-08-26T14:17:14.608 INFO:tasks.ceph.cluster1.mon.a:Sent signal 15 2023-08-26T14:17:14.609 INFO:tasks.ceph.cluster1.mgr.x:Sent signal 15 2023-08-26T14:17:14.610 INFO:tasks.ceph.cluster1.osd.0.smithi031.stderr:2023-08-26T14:17:14.606+0000 7fd376c20700 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster cluster1 -i 0 (PID: 99320) UID: 0 2023-08-26T14:17:14.610 INFO:tasks.ceph.cluster1.osd.0.smithi031.stderr:2023-08-26T14:17:14.606+0000 7fd376c20700 -1 osd.0 447 *** Got signal Terminated *** 2023-08-26T14:17:14.611 INFO:tasks.ceph.cluster1.osd.0.smithi031.stderr:2023-08-26T14:17:14.606+0000 7fd376c20700 -1 osd.0 447 *** Immediate shutdown (osd_fast_shutdown=true) *** 2023-08-26T14:17:14.612 INFO:tasks.ceph.cluster1.osd.1.smithi031.stderr:2023-08-26T14:17:14.607+0000 7f7e7383a700 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster cluster1 -i 1 (PID: 99319) UID: 0 2023-08-26T14:17:14.612 INFO:tasks.ceph.cluster1.osd.1.smithi031.stderr:2023-08-26T14:17:14.607+0000 7f7e7383a700 -1 osd.1 447 *** Got signal Terminated *** 2023-08-26T14:17:14.612 INFO:tasks.ceph.cluster1.osd.1.smithi031.stderr:2023-08-26T14:17:14.607+0000 7f7e7383a700 -1 osd.1 447 *** Immediate shutdown (osd_fast_shutdown=true) *** 2023-08-26T14:17:14.613 INFO:tasks.ceph.cluster1.osd.2.smithi031.stderr:2023-08-26T14:17:14.607+0000 7ffa54fef700 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster cluster1 -i 2 (PID: 99324) UID: 0 2023-08-26T14:17:14.613 INFO:tasks.ceph.cluster1.osd.2.smithi031.stderr:2023-08-26T14:17:14.607+0000 7ffa54fef700 -1 osd.2 447 *** Got signal Terminated *** 2023-08-26T14:17:14.613 INFO:tasks.ceph.cluster1.osd.2.smithi031.stderr:2023-08-26T14:17:14.607+0000 7ffa54fef700 -1 osd.2 447 *** Immediate shutdown (osd_fast_shutdown=true) *** 2023-08-26T14:17:14.614 INFO:tasks.ceph.cluster1.mon.a.smithi031.stderr:2023-08-26T14:17:14.608+0000 7f75d7f47700 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper kill ceph-mon -f --cluster cluster1 -i a (PID: 98962) UID: 0 2023-08-26T14:17:14.614 INFO:tasks.ceph.cluster1.mon.a.smithi031.stderr:2023-08-26T14:17:14.608+0000 7f75d7f47700 -1 mon.a@0(leader) e1 *** Got Signal Terminated *** 2023-08-26T14:17:14.614 INFO:tasks.ceph.cluster1.mgr.x.smithi031.stderr:2023-08-26T14:17:14.608+0000 7f95897ff700 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper kill ceph-mgr -f --cluster cluster1 -i x (PID: 98965) UID: 0 2023-08-26T14:17:14.615 INFO:tasks.ceph.cluster1.mgr.x.smithi031.stderr:2023-08-26T14:17:14.608+0000 7f95897ff700 -1 mgr handle_mgr_signal *** Got signal Terminated *** 2023-08-26T14:18:49.224 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:2023-08-26T14:18:49.222+0000 7f1c692a9700 0 monclient(hunting): authenticate timed out after 300 2023-08-26T14:18:49.224 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:[errno 110] RADOS timed out (error connecting to the cluster) 2023-08-26T14:18:49.225 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ CEPH_ARGS= 2023-08-26T14:18:49.225 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ ceph --cluster cluster2 service status 2023-08-26T14:23:49.329 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:2023-08-26T14:23:49.327+0000 7fad3e2ed700 0 monclient(hunting): authenticate timed out after 300 2023-08-26T14:23:49.329 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:[errno 110] RADOS timed out (error connecting to the cluster) 2023-08-26T14:23:49.331 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: 2023-08-26T14:23:49.331 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ echo 2023-08-26T14:23:49.332 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ for image_pool in ${POOL} ${PARENT_POOL} 2023-08-26T14:23:49.332 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ for image_ns in "" "${NS1}" "${NS2}" 2023-08-26T14:23:49.332 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:cluster2 mirror images 2023-08-26T14:23:49.333 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ echo 'cluster2 mirror images' 2023-08-26T14:23:49.333 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rbd --cluster cluster2 -p mirror --namespace '' ls -l 2023-08-26T14:28:49.363 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:rbd: couldn't connect to the cluster! 2023-08-26T14:28:49.363 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:rbd: listing images failed: (110) Connection timed out 2023-08-26T14:28:49.365 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ echo 2023-08-26T14:28:49.366 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: 2023-08-26T14:28:49.366 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:cluster2 mirror mirror pool info 2023-08-26T14:28:49.367 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ echo 'cluster2 mirror mirror pool info' 2023-08-26T14:28:49.367 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rbd --cluster cluster2 -p mirror --namespace '' mirror pool info 2023-08-26T14:33:49.398 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:rbd: couldn't connect to the cluster! 2023-08-26T14:33:49.400 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: 2023-08-26T14:33:49.400 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:cluster2 mirror mirror pool status 2023-08-26T14:33:49.401 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ echo 2023-08-26T14:33:49.401 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ echo 'cluster2 mirror mirror pool status' 2023-08-26T14:33:49.401 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ CEPH_ARGS= 2023-08-26T14:33:49.401 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rbd --cluster cluster2 -p mirror --namespace '' mirror pool status --verbose 2023-08-26T14:38:49.433 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:rbd: couldn't connect to the cluster! 2023-08-26T14:38:49.435 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ echo 2023-08-26T14:38:49.435 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: 2023-08-26T14:38:49.436 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ rbd --cluster cluster2 -p mirror --namespace '' ls 2023-08-26T14:43:49.472 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:cluster2 mirror rbd_mirroring omap vals 2023-08-26T14:43:49.472 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ echo 'cluster2 mirror rbd_mirroring omap vals' 2023-08-26T14:43:49.472 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rados --cluster cluster2 -p mirror --namespace '' listomapvals rbd_mirroring 2023-08-26T14:48:49.501 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:failed to fetch mon config (--no-mon-config to skip) 2023-08-26T14:48:49.502 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:cluster2 mirror rbd_mirror_leader omap vals 2023-08-26T14:48:49.502 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ echo 'cluster2 mirror rbd_mirror_leader omap vals' 2023-08-26T14:48:49.502 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rados --cluster cluster2 -p mirror --namespace '' listomapvals rbd_mirror_leader 2023-08-26T14:53:49.532 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:failed to fetch mon config (--no-mon-config to skip) 2023-08-26T14:53:49.532 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ echo 2023-08-26T14:53:49.533 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: 2023-08-26T14:53:49.533 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ for image_ns in "" "${NS1}" "${NS2}" 2023-08-26T14:53:49.534 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:cluster2 mirror ns1 images 2023-08-26T14:53:49.534 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ echo 'cluster2 mirror ns1 images' 2023-08-26T14:53:49.534 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rbd --cluster cluster2 -p mirror --namespace ns1 ls -l 2023-08-26T14:58:49.565 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:rbd: couldn't connect to the cluster! 2023-08-26T14:58:49.565 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:rbd: listing images failed: (110) Connection timed out 2023-08-26T14:58:49.567 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: 2023-08-26T14:58:49.567 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:cluster2 mirrorns1 mirror pool info 2023-08-26T14:58:49.568 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ echo 2023-08-26T14:58:49.568 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ echo 'cluster2 mirrorns1 mirror pool info' 2023-08-26T14:58:49.568 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rbd --cluster cluster2 -p mirror --namespace ns1 mirror pool info 2023-08-26T15:03:49.600 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:rbd: couldn't connect to the cluster! 2023-08-26T15:03:49.602 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: 2023-08-26T15:03:49.602 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:cluster2 mirrorns1 mirror pool status 2023-08-26T15:03:49.602 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ echo 2023-08-26T15:03:49.603 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ echo 'cluster2 mirrorns1 mirror pool status' 2023-08-26T15:03:49.603 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ CEPH_ARGS= 2023-08-26T15:03:49.603 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rbd --cluster cluster2 -p mirror --namespace ns1 mirror pool status --verbose 2023-08-26T15:08:49.634 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:rbd: couldn't connect to the cluster! 2023-08-26T15:08:49.637 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ echo 2023-08-26T15:08:49.637 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: 2023-08-26T15:08:49.638 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ rbd --cluster cluster2 -p mirror --namespace ns1 ls 2023-08-26T15:13:49.671 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:cluster2 mirror ns1 rbd_mirroring omap vals 2023-08-26T15:13:49.672 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ echo 'cluster2 mirror ns1 rbd_mirroring omap vals' 2023-08-26T15:13:49.672 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rados --cluster cluster2 -p mirror --namespace ns1 listomapvals rbd_mirroring 2023-08-26T15:18:49.702 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:failed to fetch mon config (--no-mon-config to skip) 2023-08-26T15:18:49.702 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ echo 'cluster2 mirror ns1 rbd_mirror_leader omap vals' 2023-08-26T15:18:49.703 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:cluster2 mirror ns1 rbd_mirror_leader omap vals 2023-08-26T15:18:49.703 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rados --cluster cluster2 -p mirror --namespace ns1 listomapvals rbd_mirror_leader 2023-08-26T15:23:49.732 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:failed to fetch mon config (--no-mon-config to skip) 2023-08-26T15:23:49.733 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: 2023-08-26T15:23:49.733 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ echo 2023-08-26T15:23:49.734 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ for image_ns in "" "${NS1}" "${NS2}" 2023-08-26T15:23:49.734 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:cluster2 mirror ns2 images 2023-08-26T15:23:49.735 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ echo 'cluster2 mirror ns2 images' 2023-08-26T15:23:49.735 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rbd --cluster cluster2 -p mirror --namespace ns2 ls -l 2023-08-26T15:28:49.764 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:rbd: couldn't connect to the cluster! 2023-08-26T15:28:49.765 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:rbd: listing images failed: (110) Connection timed out 2023-08-26T15:28:49.766 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ echo 2023-08-26T15:28:49.767 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: 2023-08-26T15:28:49.767 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:cluster2 mirrorns2 mirror pool info 2023-08-26T15:28:49.768 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ echo 'cluster2 mirrorns2 mirror pool info' 2023-08-26T15:28:49.769 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rbd --cluster cluster2 -p mirror --namespace ns2 mirror pool info 2023-08-26T15:33:49.799 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:rbd: couldn't connect to the cluster! 2023-08-26T15:33:49.801 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ echo 2023-08-26T15:33:49.802 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: 2023-08-26T15:33:49.802 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:cluster2 mirrorns2 mirror pool status 2023-08-26T15:33:49.803 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ echo 'cluster2 mirrorns2 mirror pool status' 2023-08-26T15:33:49.803 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ CEPH_ARGS= 2023-08-26T15:33:49.803 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rbd --cluster cluster2 -p mirror --namespace ns2 mirror pool status --verbose 2023-08-26T15:38:49.833 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:rbd: couldn't connect to the cluster! 2023-08-26T15:38:49.835 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ echo 2023-08-26T15:38:49.836 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: 2023-08-26T15:38:49.836 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ rbd --cluster cluster2 -p mirror --namespace ns2 ls 2023-08-26T15:43:49.868 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:cluster2 mirror ns2 rbd_mirroring omap vals 2023-08-26T15:43:49.869 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ echo 'cluster2 mirror ns2 rbd_mirroring omap vals' 2023-08-26T15:43:49.869 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rados --cluster cluster2 -p mirror --namespace ns2 listomapvals rbd_mirroring 2023-08-26T15:48:49.897 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:failed to fetch mon config (--no-mon-config to skip) 2023-08-26T15:48:49.898 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ echo 'cluster2 mirror ns2 rbd_mirror_leader omap vals' 2023-08-26T15:48:49.898 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:cluster2 mirror ns2 rbd_mirror_leader omap vals 2023-08-26T15:48:49.899 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rados --cluster cluster2 -p mirror --namespace ns2 listomapvals rbd_mirror_leader 2023-08-26T15:53:49.927 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:failed to fetch mon config (--no-mon-config to skip) 2023-08-26T15:53:49.928 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: 2023-08-26T15:53:49.928 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ echo 2023-08-26T15:53:49.929 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ for image_pool in ${POOL} ${PARENT_POOL} 2023-08-26T15:53:49.929 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ for image_ns in "" "${NS1}" "${NS2}" 2023-08-26T15:53:49.929 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:cluster2 mirror_parent images 2023-08-26T15:53:49.930 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ echo 'cluster2 mirror_parent images' 2023-08-26T15:53:49.930 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rbd --cluster cluster2 -p mirror_parent --namespace '' ls -l 2023-08-26T15:58:49.960 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:rbd: couldn't connect to the cluster! 2023-08-26T15:58:49.980 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:rbd: listing images failed: (110) Connection timed out 2023-08-26T15:58:49.980 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ echo 2023-08-26T15:58:49.981 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: 2023-08-26T15:58:49.981 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:cluster2 mirror_parent mirror pool info 2023-08-26T15:58:49.981 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ echo 'cluster2 mirror_parent mirror pool info' 2023-08-26T15:58:49.982 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rbd --cluster cluster2 -p mirror_parent --namespace '' mirror pool info 2023-08-26T16:03:49.995 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:rbd: couldn't connect to the cluster! 2023-08-26T16:03:49.997 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ echo 2023-08-26T16:03:49.998 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: 2023-08-26T16:03:49.998 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:cluster2 mirror_parent mirror pool status 2023-08-26T16:03:49.998 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ echo 'cluster2 mirror_parent mirror pool status' 2023-08-26T16:03:49.999 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ CEPH_ARGS= 2023-08-26T16:03:49.999 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rbd --cluster cluster2 -p mirror_parent --namespace '' mirror pool status --verbose 2023-08-26T16:08:50.029 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:rbd: couldn't connect to the cluster! 2023-08-26T16:08:50.031 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ echo 2023-08-26T16:08:50.032 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: 2023-08-26T16:08:50.032 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ rbd --cluster cluster2 -p mirror_parent --namespace '' ls 2023-08-26T16:13:50.065 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:cluster2 mirror_parent rbd_mirroring omap vals 2023-08-26T16:13:50.066 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ echo 'cluster2 mirror_parent rbd_mirroring omap vals' 2023-08-26T16:13:50.066 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rados --cluster cluster2 -p mirror_parent --namespace '' listomapvals rbd_mirroring 2023-08-26T16:18:50.094 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:failed to fetch mon config (--no-mon-config to skip) 2023-08-26T16:18:50.095 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ echo 'cluster2 mirror_parent rbd_mirror_leader omap vals' 2023-08-26T16:18:50.095 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:cluster2 mirror_parent rbd_mirror_leader omap vals 2023-08-26T16:18:50.096 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rados --cluster cluster2 -p mirror_parent --namespace '' listomapvals rbd_mirror_leader 2023-08-26T16:23:50.124 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:failed to fetch mon config (--no-mon-config to skip) 2023-08-26T16:23:50.125 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ echo 2023-08-26T16:23:50.125 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: 2023-08-26T16:23:50.125 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ for image_ns in "" "${NS1}" "${NS2}" 2023-08-26T16:23:50.126 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ echo 'cluster2 mirror_parent ns1 images' 2023-08-26T16:23:50.126 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:cluster2 mirror_parent ns1 images 2023-08-26T16:23:50.126 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rbd --cluster cluster2 -p mirror_parent --namespace ns1 ls -l 2023-08-26T16:28:50.156 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:rbd: couldn't connect to the cluster! 2023-08-26T16:28:50.157 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:rbd: listing images failed: (110) Connection timed out 2023-08-26T16:28:50.159 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ echo 2023-08-26T16:28:50.159 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: 2023-08-26T16:28:50.159 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:cluster2 mirror_parentns1 mirror pool info 2023-08-26T16:28:50.160 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ echo 'cluster2 mirror_parentns1 mirror pool info' 2023-08-26T16:28:50.160 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rbd --cluster cluster2 -p mirror_parent --namespace ns1 mirror pool info 2023-08-26T16:33:50.191 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:rbd: couldn't connect to the cluster! 2023-08-26T16:33:50.194 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ echo 2023-08-26T16:33:50.194 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: 2023-08-26T16:33:50.195 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:cluster2 mirror_parentns1 mirror pool status 2023-08-26T16:33:50.195 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ echo 'cluster2 mirror_parentns1 mirror pool status' 2023-08-26T16:33:50.196 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ CEPH_ARGS= 2023-08-26T16:33:50.196 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rbd --cluster cluster2 -p mirror_parent --namespace ns1 mirror pool status --verbose 2023-08-26T16:38:50.226 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:rbd: couldn't connect to the cluster! 2023-08-26T16:38:50.228 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: 2023-08-26T16:38:50.228 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ echo 2023-08-26T16:38:50.229 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ rbd --cluster cluster2 -p mirror_parent --namespace ns1 ls 2023-08-26T16:43:50.261 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:cluster2 mirror_parent ns1 rbd_mirroring omap vals 2023-08-26T16:43:50.261 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ echo 'cluster2 mirror_parent ns1 rbd_mirroring omap vals' 2023-08-26T16:43:50.262 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rados --cluster cluster2 -p mirror_parent --namespace ns1 listomapvals rbd_mirroring 2023-08-26T16:48:50.290 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:failed to fetch mon config (--no-mon-config to skip) 2023-08-26T16:48:50.291 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ echo 'cluster2 mirror_parent ns1 rbd_mirror_leader omap vals' 2023-08-26T16:48:50.291 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:cluster2 mirror_parent ns1 rbd_mirror_leader omap vals 2023-08-26T16:48:50.291 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rados --cluster cluster2 -p mirror_parent --namespace ns1 listomapvals rbd_mirror_leader 2023-08-26T16:53:50.320 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:failed to fetch mon config (--no-mon-config to skip) 2023-08-26T16:53:50.320 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ echo 2023-08-26T16:53:50.321 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: 2023-08-26T16:53:50.321 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ for image_ns in "" "${NS1}" "${NS2}" 2023-08-26T16:53:50.322 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:cluster2 mirror_parent ns2 images 2023-08-26T16:53:50.323 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ echo 'cluster2 mirror_parent ns2 images' 2023-08-26T16:53:50.324 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rbd --cluster cluster2 -p mirror_parent --namespace ns2 ls -l 2023-08-26T16:58:50.352 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:rbd: couldn't connect to the cluster! 2023-08-26T16:58:50.353 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:rbd: listing images failed: (110) Connection timed out 2023-08-26T16:58:50.354 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ echo 2023-08-26T16:58:50.355 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: 2023-08-26T16:58:50.355 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:cluster2 mirror_parentns2 mirror pool info 2023-08-26T16:58:50.356 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ echo 'cluster2 mirror_parentns2 mirror pool info' 2023-08-26T16:58:50.356 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rbd --cluster cluster2 -p mirror_parent --namespace ns2 mirror pool info 2023-08-26T17:03:50.387 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:rbd: couldn't connect to the cluster! 2023-08-26T17:03:50.389 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ echo 2023-08-26T17:03:50.390 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: 2023-08-26T17:03:50.390 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:cluster2 mirror_parentns2 mirror pool status 2023-08-26T17:03:50.391 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ echo 'cluster2 mirror_parentns2 mirror pool status' 2023-08-26T17:03:50.391 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ CEPH_ARGS= 2023-08-26T17:03:50.391 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rbd --cluster cluster2 -p mirror_parent --namespace ns2 mirror pool status --verbose 2023-08-26T17:08:50.422 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:rbd: couldn't connect to the cluster! 2023-08-26T17:08:50.424 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ echo 2023-08-26T17:08:50.425 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: 2023-08-26T17:08:50.425 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ rbd --cluster cluster2 -p mirror_parent --namespace ns2 ls 2023-08-26T17:13:50.458 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:cluster2 mirror_parent ns2 rbd_mirroring omap vals 2023-08-26T17:13:50.459 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ echo 'cluster2 mirror_parent ns2 rbd_mirroring omap vals' 2023-08-26T17:13:50.459 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rados --cluster cluster2 -p mirror_parent --namespace ns2 listomapvals rbd_mirroring 2023-08-26T17:18:50.487 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:failed to fetch mon config (--no-mon-config to skip) 2023-08-26T17:18:50.489 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:cluster2 mirror_parent ns2 rbd_mirror_leader omap vals 2023-08-26T17:18:50.489 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ echo 'cluster2 mirror_parent ns2 rbd_mirror_leader omap vals' 2023-08-26T17:18:50.489 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rados --cluster cluster2 -p mirror_parent --namespace ns2 listomapvals rbd_mirror_leader 2023-08-26T17:23:50.517 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:failed to fetch mon config (--no-mon-config to skip) 2023-08-26T17:23:50.518 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ echo 2023-08-26T17:23:50.519 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout: 2023-08-26T17:23:50.519 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local ret 2023-08-26T17:23:50.520 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ for cluster in "${CLUSTER1}" "${CLUSTER2}" 2023-08-26T17:23:50.520 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ seq 0 3 2023-08-26T17:23:50.520 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ for instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-08-26T17:23:50.522 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ daemon_pid_file cluster1:0 2023-08-26T17:23:50.522 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local cluster=cluster1:0 2023-08-26T17:23:50.522 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local instance 2023-08-26T17:23:50.523 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ set_cluster_instance cluster1:0 cluster instance 2023-08-26T17:23:50.523 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local val=cluster1:0 2023-08-26T17:23:50.523 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local cluster_var_name=cluster 2023-08-26T17:23:50.523 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local instance_var_name=instance 2023-08-26T17:23:50.524 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ cluster=cluster1 2023-08-26T17:23:50.524 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ instance=0 2023-08-26T17:23:50.524 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ '[' 0 = cluster1:0 ']' 2023-08-26T17:23:50.524 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ eval cluster=cluster1 2023-08-26T17:23:50.524 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ cluster=cluster1 2023-08-26T17:23:50.525 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ eval instance=0 2023-08-26T17:23:50.525 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ instance=0 2023-08-26T17:23:50.525 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'pid file' 2023-08-26T17:23:50.544 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.pid 2023-08-26T17:23:50.544 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pid_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.pid 2023-08-26T17:23:50.545 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ '[' '!' -e /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.pid ']' 2023-08-26T17:23:50.545 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ echo 'cluster1 rbd-mirror not running or unknown' '(/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.pid not exist)' 2023-08-26T17:23:50.545 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:cluster1 rbd-mirror not running or unknown (/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.pid not exist) 2023-08-26T17:23:50.546 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ continue 2023-08-26T17:23:50.546 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ for instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-08-26T17:23:50.547 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ daemon_pid_file cluster1:1 2023-08-26T17:23:50.547 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local cluster=cluster1:1 2023-08-26T17:23:50.547 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local instance 2023-08-26T17:23:50.547 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ set_cluster_instance cluster1:1 cluster instance 2023-08-26T17:23:50.547 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local val=cluster1:1 2023-08-26T17:23:50.549 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local cluster_var_name=cluster 2023-08-26T17:23:50.549 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local instance_var_name=instance 2023-08-26T17:23:50.549 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ cluster=cluster1 2023-08-26T17:23:50.549 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ instance=1 2023-08-26T17:23:50.549 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ '[' 1 = cluster1:1 ']' 2023-08-26T17:23:50.550 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ eval cluster=cluster1 2023-08-26T17:23:50.550 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ cluster=cluster1 2023-08-26T17:23:50.550 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ eval instance=1 2023-08-26T17:23:50.550 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ instance=1 2023-08-26T17:23:50.550 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.1 'pid file' 2023-08-26T17:23:50.568 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.pid 2023-08-26T17:23:50.568 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pid_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.pid 2023-08-26T17:23:50.568 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ '[' '!' -e /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.pid ']' 2023-08-26T17:23:50.568 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ echo 'cluster1 rbd-mirror not running or unknown' '(/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.pid not exist)' 2023-08-26T17:23:50.569 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:cluster1 rbd-mirror not running or unknown (/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.pid not exist) 2023-08-26T17:23:50.571 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ continue 2023-08-26T17:23:50.571 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ for instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-08-26T17:23:50.571 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ daemon_pid_file cluster1:2 2023-08-26T17:23:50.571 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local cluster=cluster1:2 2023-08-26T17:23:50.571 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local instance 2023-08-26T17:23:50.572 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ set_cluster_instance cluster1:2 cluster instance 2023-08-26T17:23:50.572 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local val=cluster1:2 2023-08-26T17:23:50.572 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local cluster_var_name=cluster 2023-08-26T17:23:50.573 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local instance_var_name=instance 2023-08-26T17:23:50.573 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ cluster=cluster1 2023-08-26T17:23:50.573 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ instance=2 2023-08-26T17:23:50.573 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ '[' 2 = cluster1:2 ']' 2023-08-26T17:23:50.573 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ eval cluster=cluster1 2023-08-26T17:23:50.574 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ cluster=cluster1 2023-08-26T17:23:50.574 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ eval instance=2 2023-08-26T17:23:50.574 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ instance=2 2023-08-26T17:23:50.574 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.2 'pid file' 2023-08-26T17:23:50.591 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.pid 2023-08-26T17:23:50.592 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pid_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.pid 2023-08-26T17:23:50.592 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ '[' '!' -e /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.pid ']' 2023-08-26T17:23:50.592 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ echo 'cluster1 rbd-mirror not running or unknown' '(/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.pid not exist)' 2023-08-26T17:23:50.592 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:cluster1 rbd-mirror not running or unknown (/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.pid not exist) 2023-08-26T17:23:50.594 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ continue 2023-08-26T17:23:50.594 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ for instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-08-26T17:23:50.595 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ daemon_pid_file cluster1:3 2023-08-26T17:23:50.595 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local cluster=cluster1:3 2023-08-26T17:23:50.595 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local instance 2023-08-26T17:23:50.595 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ set_cluster_instance cluster1:3 cluster instance 2023-08-26T17:23:50.595 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local val=cluster1:3 2023-08-26T17:23:50.596 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local cluster_var_name=cluster 2023-08-26T17:23:50.596 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local instance_var_name=instance 2023-08-26T17:23:50.596 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ cluster=cluster1 2023-08-26T17:23:50.596 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ instance=3 2023-08-26T17:23:50.596 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ '[' 3 = cluster1:3 ']' 2023-08-26T17:23:50.596 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ eval cluster=cluster1 2023-08-26T17:23:50.597 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ cluster=cluster1 2023-08-26T17:23:50.597 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ eval instance=3 2023-08-26T17:23:50.597 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ instance=3 2023-08-26T17:23:50.597 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.3 'pid file' 2023-08-26T17:23:50.614 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.pid 2023-08-26T17:23:50.615 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pid_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.pid 2023-08-26T17:23:50.615 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ '[' '!' -e /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.pid ']' 2023-08-26T17:23:50.615 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ echo 'cluster1 rbd-mirror not running or unknown' '(/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.pid not exist)' 2023-08-26T17:23:50.615 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:cluster1 rbd-mirror not running or unknown (/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.pid not exist) 2023-08-26T17:23:50.617 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ continue 2023-08-26T17:23:50.618 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ for cluster in "${CLUSTER1}" "${CLUSTER2}" 2023-08-26T17:23:50.618 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ seq 0 3 2023-08-26T17:23:50.618 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ for instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-08-26T17:23:50.618 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ daemon_pid_file cluster2:0 2023-08-26T17:23:50.618 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local cluster=cluster2:0 2023-08-26T17:23:50.619 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local instance 2023-08-26T17:23:50.619 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ set_cluster_instance cluster2:0 cluster instance 2023-08-26T17:23:50.619 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local val=cluster2:0 2023-08-26T17:23:50.619 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local cluster_var_name=cluster 2023-08-26T17:23:50.619 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local instance_var_name=instance 2023-08-26T17:23:50.620 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ cluster=cluster2 2023-08-26T17:23:50.620 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ instance=0 2023-08-26T17:23:50.620 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ '[' 0 = cluster2:0 ']' 2023-08-26T17:23:50.620 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ eval cluster=cluster2 2023-08-26T17:23:50.620 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ cluster=cluster2 2023-08-26T17:23:50.621 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ eval instance=0 2023-08-26T17:23:50.621 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ instance=0 2023-08-26T17:23:50.621 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ ceph-conf --cluster cluster2 --name client.mirror.0 'pid file' 2023-08-26T17:23:50.639 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.pid 2023-08-26T17:23:50.640 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pid_file=/var/run/ceph/rbd-mirror.cluster2-client.mirror.0.pid 2023-08-26T17:23:50.640 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ '[' '!' -e /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.pid ']' 2023-08-26T17:23:50.640 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ echo 'cluster2 rbd-mirror not running or unknown' '(/var/run/ceph/rbd-mirror.cluster2-client.mirror.0.pid not exist)' 2023-08-26T17:23:50.640 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:cluster2 rbd-mirror not running or unknown (/var/run/ceph/rbd-mirror.cluster2-client.mirror.0.pid not exist) 2023-08-26T17:23:50.642 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ continue 2023-08-26T17:23:50.642 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ for instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-08-26T17:23:50.643 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ daemon_pid_file cluster2:1 2023-08-26T17:23:50.643 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local cluster=cluster2:1 2023-08-26T17:23:50.643 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local instance 2023-08-26T17:23:50.643 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ set_cluster_instance cluster2:1 cluster instance 2023-08-26T17:23:50.643 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local val=cluster2:1 2023-08-26T17:23:50.644 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local cluster_var_name=cluster 2023-08-26T17:23:50.644 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local instance_var_name=instance 2023-08-26T17:23:50.644 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ cluster=cluster2 2023-08-26T17:23:50.644 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ instance=1 2023-08-26T17:23:50.644 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ '[' 1 = cluster2:1 ']' 2023-08-26T17:23:50.645 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ eval cluster=cluster2 2023-08-26T17:23:50.645 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ cluster=cluster2 2023-08-26T17:23:50.645 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ eval instance=1 2023-08-26T17:23:50.645 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ instance=1 2023-08-26T17:23:50.645 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ ceph-conf --cluster cluster2 --name client.mirror.1 'pid file' 2023-08-26T17:23:50.662 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.1.pid 2023-08-26T17:23:50.663 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pid_file=/var/run/ceph/rbd-mirror.cluster2-client.mirror.1.pid 2023-08-26T17:23:50.663 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ '[' '!' -e /var/run/ceph/rbd-mirror.cluster2-client.mirror.1.pid ']' 2023-08-26T17:23:50.663 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ echo 'cluster2 rbd-mirror not running or unknown' '(/var/run/ceph/rbd-mirror.cluster2-client.mirror.1.pid not exist)' 2023-08-26T17:23:50.664 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:cluster2 rbd-mirror not running or unknown (/var/run/ceph/rbd-mirror.cluster2-client.mirror.1.pid not exist) 2023-08-26T17:23:50.665 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ continue 2023-08-26T17:23:50.666 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ for instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-08-26T17:23:50.666 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ daemon_pid_file cluster2:2 2023-08-26T17:23:50.666 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local cluster=cluster2:2 2023-08-26T17:23:50.666 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local instance 2023-08-26T17:23:50.666 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ set_cluster_instance cluster2:2 cluster instance 2023-08-26T17:23:50.667 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local val=cluster2:2 2023-08-26T17:23:50.667 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local cluster_var_name=cluster 2023-08-26T17:23:50.667 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local instance_var_name=instance 2023-08-26T17:23:50.667 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ cluster=cluster2 2023-08-26T17:23:50.667 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ instance=2 2023-08-26T17:23:50.668 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ '[' 2 = cluster2:2 ']' 2023-08-26T17:23:50.668 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ eval cluster=cluster2 2023-08-26T17:23:50.668 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ cluster=cluster2 2023-08-26T17:23:50.668 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ eval instance=2 2023-08-26T17:23:50.668 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ instance=2 2023-08-26T17:23:50.669 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ ceph-conf --cluster cluster2 --name client.mirror.2 'pid file' 2023-08-26T17:23:50.686 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.2.pid 2023-08-26T17:23:50.686 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pid_file=/var/run/ceph/rbd-mirror.cluster2-client.mirror.2.pid 2023-08-26T17:23:50.686 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ '[' '!' -e /var/run/ceph/rbd-mirror.cluster2-client.mirror.2.pid ']' 2023-08-26T17:23:50.687 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ echo 'cluster2 rbd-mirror not running or unknown' '(/var/run/ceph/rbd-mirror.cluster2-client.mirror.2.pid not exist)' 2023-08-26T17:23:50.687 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:cluster2 rbd-mirror not running or unknown (/var/run/ceph/rbd-mirror.cluster2-client.mirror.2.pid not exist) 2023-08-26T17:23:50.688 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ continue 2023-08-26T17:23:50.689 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ for instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-08-26T17:23:50.689 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ daemon_pid_file cluster2:3 2023-08-26T17:23:50.689 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local cluster=cluster2:3 2023-08-26T17:23:50.689 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local instance 2023-08-26T17:23:50.690 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ set_cluster_instance cluster2:3 cluster instance 2023-08-26T17:23:50.690 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local val=cluster2:3 2023-08-26T17:23:50.690 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local cluster_var_name=cluster 2023-08-26T17:23:50.690 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ local instance_var_name=instance 2023-08-26T17:23:50.690 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ cluster=cluster2 2023-08-26T17:23:50.690 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ instance=3 2023-08-26T17:23:50.691 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ '[' 3 = cluster2:3 ']' 2023-08-26T17:23:50.691 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ eval cluster=cluster2 2023-08-26T17:23:50.691 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ cluster=cluster2 2023-08-26T17:23:50.691 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ eval instance=3 2023-08-26T17:23:50.691 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ instance=3 2023-08-26T17:23:50.692 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+++ ceph-conf --cluster cluster2 --name client.mirror.3 'pid file' 2023-08-26T17:23:50.709 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.3.pid 2023-08-26T17:23:50.709 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local pid_file=/var/run/ceph/rbd-mirror.cluster2-client.mirror.3.pid 2023-08-26T17:23:50.710 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ '[' '!' -e /var/run/ceph/rbd-mirror.cluster2-client.mirror.3.pid ']' 2023-08-26T17:23:50.710 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ echo 'cluster2 rbd-mirror not running or unknown' '(/var/run/ceph/rbd-mirror.cluster2-client.mirror.3.pid not exist)' 2023-08-26T17:23:50.710 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:cluster2 rbd-mirror not running or unknown (/var/run/ceph/rbd-mirror.cluster2-client.mirror.3.pid not exist) 2023-08-26T17:23:50.711 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ continue 2023-08-26T17:23:50.712 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ return 2023-08-26T17:23:50.712 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ '[' -z '' ']' 2023-08-26T17:23:50.712 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster instance 2023-08-26T17:23:50.713 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ CEPH_ARGS= 2023-08-26T17:23:50.713 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ ceph --cluster cluster1 osd pool rm mirror mirror --yes-i-really-really-mean-it 2023-08-26T17:28:50.819 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:2023-08-26T17:28:50.818+0000 7f3a4a72d700 0 monclient(hunting): authenticate timed out after 300 2023-08-26T17:28:50.820 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:[errno 110] RADOS timed out (error connecting to the cluster) 2023-08-26T17:28:50.821 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ CEPH_ARGS= 2023-08-26T17:28:50.821 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ ceph --cluster cluster2 osd pool rm mirror mirror --yes-i-really-really-mean-it 2023-08-26T17:33:50.926 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:2023-08-26T17:33:50.926+0000 7fae8957e700 0 monclient(hunting): authenticate timed out after 300 2023-08-26T17:33:50.927 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:[errno 110] RADOS timed out (error connecting to the cluster) 2023-08-26T17:33:50.928 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ CEPH_ARGS= 2023-08-26T17:33:50.929 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ ceph --cluster cluster1 osd pool rm mirror_parent mirror_parent --yes-i-really-really-mean-it 2023-08-26T17:38:51.033 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:2023-08-26T17:38:51.032+0000 7f7a51c66700 0 monclient(hunting): authenticate timed out after 300 2023-08-26T17:38:51.034 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:[errno 110] RADOS timed out (error connecting to the cluster) 2023-08-26T17:38:51.034 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ CEPH_ARGS= 2023-08-26T17:38:51.034 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ ceph --cluster cluster2 osd pool rm mirror_parent mirror_parent --yes-i-really-really-mean-it 2023-08-26T17:43:51.140 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:2023-08-26T17:43:51.139+0000 7fecbd1cc700 0 monclient(hunting): authenticate timed out after 300 2023-08-26T17:43:51.140 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:[errno 110] RADOS timed out (error connecting to the cluster) 2023-08-26T17:43:51.142 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ for cluster in "${CLUSTER1}" "${CLUSTER2}" 2023-08-26T17:43:51.142 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ stop_mirrors cluster1 2023-08-26T17:43:51.142 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster1 2023-08-26T17:43:51.142 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local sig= 2023-08-26T17:43:51.143 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ seq 0 3 2023-08-26T17:43:51.147 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ for instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-08-26T17:43:51.147 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ stop_mirror cluster1:0 '' 2023-08-26T17:43:51.148 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster1:0 2023-08-26T17:43:51.148 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local sig= 2023-08-26T17:43:51.148 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ test -n 1 2023-08-26T17:43:51.148 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ return 2023-08-26T17:43:51.148 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ for instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-08-26T17:43:51.149 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ stop_mirror cluster1:1 '' 2023-08-26T17:43:51.149 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster1:1 2023-08-26T17:43:51.149 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local sig= 2023-08-26T17:43:51.149 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ test -n 1 2023-08-26T17:43:51.149 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ return 2023-08-26T17:43:51.150 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ for instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-08-26T17:43:51.150 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ stop_mirror cluster1:2 '' 2023-08-26T17:43:51.150 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster1:2 2023-08-26T17:43:51.150 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local sig= 2023-08-26T17:43:51.150 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ test -n 1 2023-08-26T17:43:51.151 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ return 2023-08-26T17:43:51.151 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ for instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-08-26T17:43:51.151 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ stop_mirror cluster1:3 '' 2023-08-26T17:43:51.151 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster1:3 2023-08-26T17:43:51.151 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local sig= 2023-08-26T17:43:51.151 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ test -n 1 2023-08-26T17:43:51.152 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ return 2023-08-26T17:43:51.152 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ for cluster in "${CLUSTER1}" "${CLUSTER2}" 2023-08-26T17:43:51.152 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ stop_mirrors cluster2 2023-08-26T17:43:51.152 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster2 2023-08-26T17:43:51.152 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local sig= 2023-08-26T17:43:51.153 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:++ seq 0 3 2023-08-26T17:43:51.153 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ for instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-08-26T17:43:51.153 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ stop_mirror cluster2:0 '' 2023-08-26T17:43:51.153 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster2:0 2023-08-26T17:43:51.153 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local sig= 2023-08-26T17:43:51.154 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ test -n 1 2023-08-26T17:43:51.154 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ return 2023-08-26T17:43:51.154 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ for instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-08-26T17:43:51.154 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ stop_mirror cluster2:1 '' 2023-08-26T17:43:51.154 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster2:1 2023-08-26T17:43:51.154 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local sig= 2023-08-26T17:43:51.155 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ test -n 1 2023-08-26T17:43:51.155 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ return 2023-08-26T17:43:51.155 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ for instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-08-26T17:43:51.155 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ stop_mirror cluster2:2 '' 2023-08-26T17:43:51.155 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster2:2 2023-08-26T17:43:51.156 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local sig= 2023-08-26T17:43:51.156 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ test -n 1 2023-08-26T17:43:51.156 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ return 2023-08-26T17:43:51.156 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ for instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2023-08-26T17:43:51.156 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ stop_mirror cluster2:3 '' 2023-08-26T17:43:51.157 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local cluster=cluster2:3 2023-08-26T17:43:51.157 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ local sig= 2023-08-26T17:43:51.157 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ test -n 1 2023-08-26T17:43:51.157 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ return 2023-08-26T17:43:51.157 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ '[' -z 1 ']' 2023-08-26T17:43:51.158 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ test '' = /tmp/tmp.TfDn6BcQ3N 2023-08-26T17:43:51.158 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ rm -Rf /tmp/tmp.TfDn6BcQ3N 2023-08-26T17:43:51.158 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ '[' 1 -eq 0 ']' 2023-08-26T17:43:51.158 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ echo FAIL 2023-08-26T17:43:51.159 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stdout:FAIL 2023-08-26T17:43:51.160 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-08-26T17:43:51.160 INFO:tasks.workunit.cluster1.client.mirror.smithi124.stderr:+ exit 1 2023-08-26T17:43:51.160 INFO:tasks.workunit:Stopping ['rbd/rbd_mirror_stress.sh'] on cluster1.client.mirror... 2023-08-26T17:43:51.161 DEBUG:teuthology.orchestra.run.smithi124:> sudo rm -rf -- /home/ubuntu/cephtest/workunits.list.cluster1.client.mirror /home/ubuntu/cephtest/clone.cluster1.client.mirror 2023-08-26T17:43:51.440 ERROR:teuthology.run_tasks:Saw exception from tasks. Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_449a1bc2027504e7b3c3d7b30fa4178906581da7/teuthology/run_tasks.py", line 105, in run_tasks manager = run_one_task(taskname, ctx=ctx, config=config) File "/home/teuthworker/src/git.ceph.com_teuthology_449a1bc2027504e7b3c3d7b30fa4178906581da7/teuthology/run_tasks.py", line 83, in run_one_task return task(**kwargs) File "/home/teuthworker/src/github.com_ceph_ceph-c_e7eb9853fddaf661c68f207346131a7e00d8ed2f/qa/tasks/workunit.py", line 129, in task p.spawn(_run_tests, ctx, refspec, role, tests, File "/home/teuthworker/src/git.ceph.com_teuthology_449a1bc2027504e7b3c3d7b30fa4178906581da7/teuthology/parallel.py", line 84, in __exit__ for result in self: File "/home/teuthworker/src/git.ceph.com_teuthology_449a1bc2027504e7b3c3d7b30fa4178906581da7/teuthology/parallel.py", line 98, in __next__ resurrect_traceback(result) File "/home/teuthworker/src/git.ceph.com_teuthology_449a1bc2027504e7b3c3d7b30fa4178906581da7/teuthology/parallel.py", line 30, in resurrect_traceback raise exc.exc_info[1] File "/home/teuthworker/src/git.ceph.com_teuthology_449a1bc2027504e7b3c3d7b30fa4178906581da7/teuthology/parallel.py", line 23, in capture_traceback return func(*args, **kwargs) File "/home/teuthworker/src/github.com_ceph_ceph-c_e7eb9853fddaf661c68f207346131a7e00d8ed2f/qa/tasks/workunit.py", line 424, in _run_tests remote.run( File "/home/teuthworker/src/git.ceph.com_teuthology_449a1bc2027504e7b3c3d7b30fa4178906581da7/teuthology/orchestra/remote.py", line 522, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_449a1bc2027504e7b3c3d7b30fa4178906581da7/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_449a1bc2027504e7b3c3d7b30fa4178906581da7/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_449a1bc2027504e7b3c3d7b30fa4178906581da7/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed (workunit test rbd/rbd_mirror_stress.sh) on smithi124 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.cluster1.mirror/client.mirror/tmp && cd -- /home/ubuntu/cephtest/mnt.cluster1.mirror/client.mirror/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=e7eb9853fddaf661c68f207346131a7e00d8ed2f TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster cluster1" CEPH_ID="mirror" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.cluster1.client.mirror CEPH_ROOT=/home/ubuntu/cephtest/clone.cluster1.client.mirror CEPH_MNT=/home/ubuntu/cephtest/mnt.cluster1.mirror CEPH_ARGS=\'\' MIRROR_IMAGE_MODE=snapshot MIRROR_POOL_MODE=image RBD_IMAGE_FEATURES=layering,exclusive-lock RBD_MIRROR_INSTANCES=4 RBD_MIRROR_USE_EXISTING_CLUSTER=1 RBD_MIRROR_USE_RBD_MIRROR=1 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 6h /home/ubuntu/cephtest/clone.cluster1.client.mirror/qa/workunits/rbd/rbd_mirror_stress.sh' 2023-08-26T17:43:51.505 ERROR:teuthology.util.sentry: Sentry event: https://sentry.ceph.com/organizations/ceph/?query=6f5588fc58b2415f80c823da166634a5 Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_449a1bc2027504e7b3c3d7b30fa4178906581da7/teuthology/run_tasks.py", line 105, in run_tasks manager = run_one_task(taskname, ctx=ctx, config=config) File "/home/teuthworker/src/git.ceph.com_teuthology_449a1bc2027504e7b3c3d7b30fa4178906581da7/teuthology/run_tasks.py", line 83, in run_one_task return task(**kwargs) File "/home/teuthworker/src/github.com_ceph_ceph-c_e7eb9853fddaf661c68f207346131a7e00d8ed2f/qa/tasks/workunit.py", line 129, in task p.spawn(_run_tests, ctx, refspec, role, tests, File "/home/teuthworker/src/git.ceph.com_teuthology_449a1bc2027504e7b3c3d7b30fa4178906581da7/teuthology/parallel.py", line 84, in __exit__ for result in self: File "/home/teuthworker/src/git.ceph.com_teuthology_449a1bc2027504e7b3c3d7b30fa4178906581da7/teuthology/parallel.py", line 98, in __next__ resurrect_traceback(result) File "/home/teuthworker/src/git.ceph.com_teuthology_449a1bc2027504e7b3c3d7b30fa4178906581da7/teuthology/parallel.py", line 30, in resurrect_traceback raise exc.exc_info[1] File "/home/teuthworker/src/git.ceph.com_teuthology_449a1bc2027504e7b3c3d7b30fa4178906581da7/teuthology/parallel.py", line 23, in capture_traceback return func(*args, **kwargs) File "/home/teuthworker/src/github.com_ceph_ceph-c_e7eb9853fddaf661c68f207346131a7e00d8ed2f/qa/tasks/workunit.py", line 424, in _run_tests remote.run( File "/home/teuthworker/src/git.ceph.com_teuthology_449a1bc2027504e7b3c3d7b30fa4178906581da7/teuthology/orchestra/remote.py", line 522, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_449a1bc2027504e7b3c3d7b30fa4178906581da7/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_449a1bc2027504e7b3c3d7b30fa4178906581da7/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_449a1bc2027504e7b3c3d7b30fa4178906581da7/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed (workunit test rbd/rbd_mirror_stress.sh) on smithi124 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.cluster1.mirror/client.mirror/tmp && cd -- /home/ubuntu/cephtest/mnt.cluster1.mirror/client.mirror/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=e7eb9853fddaf661c68f207346131a7e00d8ed2f TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster cluster1" CEPH_ID="mirror" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.cluster1.client.mirror CEPH_ROOT=/home/ubuntu/cephtest/clone.cluster1.client.mirror CEPH_MNT=/home/ubuntu/cephtest/mnt.cluster1.mirror CEPH_ARGS=\'\' MIRROR_IMAGE_MODE=snapshot MIRROR_POOL_MODE=image RBD_IMAGE_FEATURES=layering,exclusive-lock RBD_MIRROR_INSTANCES=4 RBD_MIRROR_USE_EXISTING_CLUSTER=1 RBD_MIRROR_USE_RBD_MIRROR=1 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 6h /home/ubuntu/cephtest/clone.cluster1.client.mirror/qa/workunits/rbd/rbd_mirror_stress.sh' 2023-08-26T17:43:51.508 DEBUG:teuthology.run_tasks:Unwinding manager rbd-mirror-thrash 2023-08-26T17:43:51.519 INFO:tasks.rbd_mirror_thrash:joining rbd_mirror_thrash 2023-08-26T17:43:51.520 ERROR:teuthology.run_tasks:Manager failed: rbd-mirror-thrash Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_449a1bc2027504e7b3c3d7b30fa4178906581da7/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ next(self.gen) File "/home/teuthworker/src/github.com_ceph_ceph-c_e7eb9853fddaf661c68f207346131a7e00d8ed2f/qa/tasks/rbd_mirror_thrash.py", line 216, in task raise RuntimeError('error during thrashing') RuntimeError: error during thrashing 2023-08-26T17:43:51.539 DEBUG:teuthology.run_tasks:Unwinding manager rbd-mirror-thrash 2023-08-26T17:43:51.549 INFO:tasks.rbd_mirror_thrash:joining rbd_mirror_thrash 2023-08-26T17:43:51.549 ERROR:teuthology.run_tasks:Manager failed: rbd-mirror-thrash Traceback (most recent call last): File "/home/teuthworker/src/github.com_ceph_ceph-c_e7eb9853fddaf661c68f207346131a7e00d8ed2f/qa/tasks/rbd_mirror_thrash.py", line 211, in task yield File "/home/teuthworker/src/git.ceph.com_teuthology_449a1bc2027504e7b3c3d7b30fa4178906581da7/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ next(self.gen) File "/home/teuthworker/src/github.com_ceph_ceph-c_e7eb9853fddaf661c68f207346131a7e00d8ed2f/qa/tasks/rbd_mirror_thrash.py", line 216, in task raise RuntimeError('error during thrashing') RuntimeError: error during thrashing During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_449a1bc2027504e7b3c3d7b30fa4178906581da7/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/github.com_ceph_ceph-c_e7eb9853fddaf661c68f207346131a7e00d8ed2f/qa/tasks/rbd_mirror_thrash.py", line 216, in task raise RuntimeError('error during thrashing') RuntimeError: error during thrashing 2023-08-26T17:43:51.554 DEBUG:teuthology.run_tasks:Unwinding manager rbd-mirror 2023-08-26T17:43:51.565 DEBUG:tasks.rbd_mirror.cluster2.client.mirror.3:waiting for process to exit 2023-08-26T17:43:51.565 INFO:teuthology.orchestra.run:waiting for 300 2023-08-26T17:43:51.567 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi124.stderr:2023-08-26T17:43:51.566+0000 7fc24c3d8700 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.3 (PID: 118321) UID: 1000 2023-08-26T17:48:50.628 ERROR:teuthology.run_tasks:Manager failed: rbd-mirror Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_449a1bc2027504e7b3c3d7b30fa4178906581da7/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/home/teuthworker/src/git.ceph.com_teuthology_449a1bc2027504e7b3c3d7b30fa4178906581da7/teuthology/task/__init__.py", line 132, in __exit__ self.end() File "/home/teuthworker/src/github.com_ceph_ceph-c_e7eb9853fddaf661c68f207346131a7e00d8ed2f/qa/tasks/rbd_mirror.py", line 117, in end mirror_daemon.stop() File "/home/teuthworker/src/git.ceph.com_teuthology_449a1bc2027504e7b3c3d7b30fa4178906581da7/teuthology/orchestra/daemon/state.py", line 139, in stop run.wait([self.proc], timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_449a1bc2027504e7b3c3d7b30fa4178906581da7/teuthology/orchestra/run.py", line 473, in wait check_time() File "/home/teuthworker/src/git.ceph.com_teuthology_449a1bc2027504e7b3c3d7b30fa4178906581da7/teuthology/contextutil.py", line 134, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (51) after waiting for 300 seconds 2023-08-26T17:48:50.642 DEBUG:teuthology.run_tasks:Unwinding manager rbd-mirror 2023-08-26T17:48:50.654 DEBUG:tasks.rbd_mirror.cluster2.client.mirror.2:waiting for process to exit 2023-08-26T17:48:50.654 INFO:teuthology.orchestra.run:waiting for 300 2023-08-26T17:48:50.658 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi124.stderr:2023-08-26T17:48:50.654+0000 7fdfe8197700 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.2 (PID: 141730) UID: 1000 2023-08-26T17:53:49.720 ERROR:teuthology.run_tasks:Manager failed: rbd-mirror Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_449a1bc2027504e7b3c3d7b30fa4178906581da7/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/home/teuthworker/src/git.ceph.com_teuthology_449a1bc2027504e7b3c3d7b30fa4178906581da7/teuthology/task/__init__.py", line 132, in __exit__ self.end() File "/home/teuthworker/src/github.com_ceph_ceph-c_e7eb9853fddaf661c68f207346131a7e00d8ed2f/qa/tasks/rbd_mirror.py", line 117, in end mirror_daemon.stop() File "/home/teuthworker/src/git.ceph.com_teuthology_449a1bc2027504e7b3c3d7b30fa4178906581da7/teuthology/orchestra/daemon/state.py", line 139, in stop run.wait([self.proc], timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_449a1bc2027504e7b3c3d7b30fa4178906581da7/teuthology/orchestra/run.py", line 473, in wait check_time() File "/home/teuthworker/src/git.ceph.com_teuthology_449a1bc2027504e7b3c3d7b30fa4178906581da7/teuthology/contextutil.py", line 134, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (51) after waiting for 300 seconds 2023-08-26T17:53:49.724 DEBUG:teuthology.run_tasks:Unwinding manager rbd-mirror 2023-08-26T17:53:49.735 DEBUG:tasks.rbd_mirror.cluster2.client.mirror.1:waiting for process to exit 2023-08-26T17:53:49.736 INFO:teuthology.orchestra.run:waiting for 300 2023-08-26T17:53:49.739 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi124.stderr:2023-08-26T17:53:49.736+0000 7fee5f5b3700 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.1 (PID: 140984) UID: 1000 2023-08-26T17:58:48.795 ERROR:teuthology.run_tasks:Manager failed: rbd-mirror Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_449a1bc2027504e7b3c3d7b30fa4178906581da7/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/home/teuthworker/src/git.ceph.com_teuthology_449a1bc2027504e7b3c3d7b30fa4178906581da7/teuthology/task/__init__.py", line 132, in __exit__ self.end() File "/home/teuthworker/src/github.com_ceph_ceph-c_e7eb9853fddaf661c68f207346131a7e00d8ed2f/qa/tasks/rbd_mirror.py", line 117, in end mirror_daemon.stop() File "/home/teuthworker/src/git.ceph.com_teuthology_449a1bc2027504e7b3c3d7b30fa4178906581da7/teuthology/orchestra/daemon/state.py", line 139, in stop run.wait([self.proc], timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_449a1bc2027504e7b3c3d7b30fa4178906581da7/teuthology/orchestra/run.py", line 473, in wait check_time() File "/home/teuthworker/src/git.ceph.com_teuthology_449a1bc2027504e7b3c3d7b30fa4178906581da7/teuthology/contextutil.py", line 134, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (51) after waiting for 300 seconds 2023-08-26T17:58:48.799 DEBUG:teuthology.run_tasks:Unwinding manager rbd-mirror 2023-08-26T17:58:48.810 ERROR:tasks.rbd_mirror.cluster2.client.mirror.0:tried to stop a non-running daemon 2023-08-26T17:58:48.813 DEBUG:teuthology.run_tasks:Unwinding manager rbd-mirror 2023-08-26T17:58:48.825 DEBUG:tasks.rbd_mirror.cluster1.client.mirror.3:waiting for process to exit 2023-08-26T17:58:48.825 INFO:teuthology.orchestra.run:waiting for 300 2023-08-26T17:58:48.827 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi124.stderr:2023-08-26T17:58:48.824+0000 7f5fb37a9700 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.3 (PID: 124601) UID: 1000 2023-08-26T18:03:47.891 ERROR:teuthology.run_tasks:Manager failed: rbd-mirror Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_449a1bc2027504e7b3c3d7b30fa4178906581da7/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/home/teuthworker/src/git.ceph.com_teuthology_449a1bc2027504e7b3c3d7b30fa4178906581da7/teuthology/task/__init__.py", line 132, in __exit__ self.end() File "/home/teuthworker/src/github.com_ceph_ceph-c_e7eb9853fddaf661c68f207346131a7e00d8ed2f/qa/tasks/rbd_mirror.py", line 117, in end mirror_daemon.stop() File "/home/teuthworker/src/git.ceph.com_teuthology_449a1bc2027504e7b3c3d7b30fa4178906581da7/teuthology/orchestra/daemon/state.py", line 139, in stop run.wait([self.proc], timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_449a1bc2027504e7b3c3d7b30fa4178906581da7/teuthology/orchestra/run.py", line 473, in wait check_time() File "/home/teuthworker/src/git.ceph.com_teuthology_449a1bc2027504e7b3c3d7b30fa4178906581da7/teuthology/contextutil.py", line 134, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (51) after waiting for 300 seconds 2023-08-26T18:03:47.895 DEBUG:teuthology.run_tasks:Unwinding manager rbd-mirror 2023-08-26T18:03:47.908 ERROR:tasks.rbd_mirror.cluster1.client.mirror.2:tried to stop a non-running daemon 2023-08-26T18:03:47.912 DEBUG:teuthology.run_tasks:Unwinding manager rbd-mirror 2023-08-26T18:03:47.947 DEBUG:tasks.rbd_mirror.cluster1.client.mirror.1:waiting for process to exit 2023-08-26T18:03:47.948 INFO:teuthology.orchestra.run:waiting for 300 2023-08-26T18:03:47.950 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi124.stderr:2023-08-26T18:03:47.948+0000 7f0346352700 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.1 (PID: 116951) UID: 1000 2023-08-26T18:08:47.015 ERROR:teuthology.run_tasks:Manager failed: rbd-mirror Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_449a1bc2027504e7b3c3d7b30fa4178906581da7/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/home/teuthworker/src/git.ceph.com_teuthology_449a1bc2027504e7b3c3d7b30fa4178906581da7/teuthology/task/__init__.py", line 132, in __exit__ self.end() File "/home/teuthworker/src/github.com_ceph_ceph-c_e7eb9853fddaf661c68f207346131a7e00d8ed2f/qa/tasks/rbd_mirror.py", line 117, in end mirror_daemon.stop() File "/home/teuthworker/src/git.ceph.com_teuthology_449a1bc2027504e7b3c3d7b30fa4178906581da7/teuthology/orchestra/daemon/state.py", line 139, in stop run.wait([self.proc], timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_449a1bc2027504e7b3c3d7b30fa4178906581da7/teuthology/orchestra/run.py", line 473, in wait check_time() File "/home/teuthworker/src/git.ceph.com_teuthology_449a1bc2027504e7b3c3d7b30fa4178906581da7/teuthology/contextutil.py", line 134, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (51) after waiting for 300 seconds 2023-08-26T18:08:47.021 DEBUG:teuthology.run_tasks:Unwinding manager rbd-mirror 2023-08-26T18:08:47.034 DEBUG:tasks.rbd_mirror.cluster1.client.mirror.0:waiting for process to exit 2023-08-26T18:08:47.034 INFO:teuthology.orchestra.run:waiting for 300 2023-08-26T18:08:47.039 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi124.stderr:2023-08-26T18:08:47.035+0000 7fbe5629e700 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.0 (PID: 141621) UID: 1000 2023-08-26T18:13:46.105 ERROR:teuthology.run_tasks:Manager failed: rbd-mirror Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_449a1bc2027504e7b3c3d7b30fa4178906581da7/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/home/teuthworker/src/git.ceph.com_teuthology_449a1bc2027504e7b3c3d7b30fa4178906581da7/teuthology/task/__init__.py", line 132, in __exit__ self.end() File "/home/teuthworker/src/github.com_ceph_ceph-c_e7eb9853fddaf661c68f207346131a7e00d8ed2f/qa/tasks/rbd_mirror.py", line 117, in end mirror_daemon.stop() File "/home/teuthworker/src/git.ceph.com_teuthology_449a1bc2027504e7b3c3d7b30fa4178906581da7/teuthology/orchestra/daemon/state.py", line 139, in stop run.wait([self.proc], timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_449a1bc2027504e7b3c3d7b30fa4178906581da7/teuthology/orchestra/run.py", line 473, in wait check_time() File "/home/teuthworker/src/git.ceph.com_teuthology_449a1bc2027504e7b3c3d7b30fa4178906581da7/teuthology/contextutil.py", line 134, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (51) after waiting for 300 seconds 2023-08-26T18:13:46.113 DEBUG:teuthology.run_tasks:Unwinding manager ceph 2023-08-26T18:13:46.126 INFO:tasks.ceph.ceph_manager.cluster2:waiting for clean 2023-08-26T18:13:46.128 DEBUG:teuthology.orchestra.run.smithi124:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2023-08-26T18:15:46.195 DEBUG:teuthology.orchestra.run:got remote process result: 124 2023-08-26T18:15:46.196 ERROR:teuthology.contextutil:Saw exception from nested tasks Traceback (most recent call last): File "/home/teuthworker/src/github.com_ceph_ceph-c_e7eb9853fddaf661c68f207346131a7e00d8ed2f/qa/tasks/ceph.py", line 1940, in task yield File "/home/teuthworker/src/git.ceph.com_teuthology_449a1bc2027504e7b3c3d7b30fa4178906581da7/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ next(self.gen) File "/home/teuthworker/src/github.com_ceph_ceph-c_e7eb9853fddaf661c68f207346131a7e00d8ed2f/qa/tasks/rbd_mirror_thrash.py", line 216, in task raise RuntimeError('error during thrashing') RuntimeError: error during thrashing During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_449a1bc2027504e7b3c3d7b30fa4178906581da7/teuthology/contextutil.py", line 32, in nested yield vars File "/home/teuthworker/src/github.com_ceph_ceph-c_e7eb9853fddaf661c68f207346131a7e00d8ed2f/qa/tasks/ceph.py", line 1950, in task ctx.managers[config['cluster']].wait_for_clean() File "/home/teuthworker/src/github.com_ceph_ceph-c_e7eb9853fddaf661c68f207346131a7e00d8ed2f/qa/tasks/ceph_manager.py", line 2742, in wait_for_clean num_active_clean = self.get_num_active_clean() File "/home/teuthworker/src/github.com_ceph_ceph-c_e7eb9853fddaf661c68f207346131a7e00d8ed2f/qa/tasks/ceph_manager.py", line 2574, in get_num_active_clean pgs = self.get_pg_stats() File "/home/teuthworker/src/github.com_ceph_ceph-c_e7eb9853fddaf661c68f207346131a7e00d8ed2f/qa/tasks/ceph_manager.py", line 2340, in get_pg_stats out = self.raw_cluster_cmd('pg', 'dump', '--format=json') File "/home/teuthworker/src/github.com_ceph_ceph-c_e7eb9853fddaf661c68f207346131a7e00d8ed2f/qa/tasks/ceph_manager.py", line 1595, in raw_cluster_cmd return self.run_cluster_cmd(**kwargs).stdout.getvalue() File "/home/teuthworker/src/github.com_ceph_ceph-c_e7eb9853fddaf661c68f207346131a7e00d8ed2f/qa/tasks/ceph_manager.py", line 1586, in run_cluster_cmd return self.controller.run(**kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_449a1bc2027504e7b3c3d7b30fa4178906581da7/teuthology/orchestra/remote.py", line 522, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_449a1bc2027504e7b3c3d7b30fa4178906581da7/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_449a1bc2027504e7b3c3d7b30fa4178906581da7/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_449a1bc2027504e7b3c3d7b30fa4178906581da7/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi124 with status 124: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json' 2023-08-26T18:15:46.208 INFO:teuthology.misc:Shutting down mds daemons... 2023-08-26T18:15:46.209 INFO:teuthology.misc:Shutting down osd daemons... 2023-08-26T18:15:46.209 DEBUG:tasks.ceph.cluster2.osd.0:waiting for process to exit 2023-08-26T18:15:46.210 INFO:teuthology.orchestra.run:waiting for 300 2023-08-26T18:15:46.214 DEBUG:teuthology.orchestra.run:got remote process result: 5 2023-08-26T18:15:46.214 ERROR:teuthology.orchestra.daemon.state:Error while waiting for process to exit Traceback (most recent call last): File "/home/teuthworker/src/github.com_ceph_ceph-c_e7eb9853fddaf661c68f207346131a7e00d8ed2f/qa/tasks/ceph.py", line 1940, in task yield File "/home/teuthworker/src/git.ceph.com_teuthology_449a1bc2027504e7b3c3d7b30fa4178906581da7/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ next(self.gen) File "/home/teuthworker/src/github.com_ceph_ceph-c_e7eb9853fddaf661c68f207346131a7e00d8ed2f/qa/tasks/rbd_mirror_thrash.py", line 216, in task raise RuntimeError('error during thrashing') RuntimeError: error during thrashing During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/github.com_ceph_ceph-c_e7eb9853fddaf661c68f207346131a7e00d8ed2f/qa/tasks/ceph.py", line 1469, in run_daemon yield File "/home/teuthworker/src/git.ceph.com_teuthology_449a1bc2027504e7b3c3d7b30fa4178906581da7/teuthology/contextutil.py", line 32, in nested yield vars File "/home/teuthworker/src/github.com_ceph_ceph-c_e7eb9853fddaf661c68f207346131a7e00d8ed2f/qa/tasks/ceph.py", line 1950, in task ctx.managers[config['cluster']].wait_for_clean() File "/home/teuthworker/src/github.com_ceph_ceph-c_e7eb9853fddaf661c68f207346131a7e00d8ed2f/qa/tasks/ceph_manager.py", line 2742, in wait_for_clean num_active_clean = self.get_num_active_clean() File "/home/teuthworker/src/github.com_ceph_ceph-c_e7eb9853fddaf661c68f207346131a7e00d8ed2f/qa/tasks/ceph_manager.py", line 2574, in get_num_active_clean pgs = self.get_pg_stats() File "/home/teuthworker/src/github.com_ceph_ceph-c_e7eb9853fddaf661c68f207346131a7e00d8ed2f/qa/tasks/ceph_manager.py", line 2340, in get_pg_stats out = self.raw_cluster_cmd('pg', 'dump', '--format=json') File "/home/teuthworker/src/github.com_ceph_ceph-c_e7eb9853fddaf661c68f207346131a7e00d8ed2f/qa/tasks/ceph_manager.py", line 1595, in raw_cluster_cmd return self.run_cluster_cmd(**kwargs).stdout.getvalue() File "/home/teuthworker/src/github.com_ceph_ceph-c_e7eb9853fddaf661c68f207346131a7e00d8ed2f/qa/tasks/ceph_manager.py", line 1586, in run_cluster_cmd return self.controller.run(**kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_449a1bc2027504e7b3c3d7b30fa4178906581da7/teuthology/orchestra/remote.py", line 522, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_449a1bc2027504e7b3c3d7b30fa4178906581da7/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_449a1bc2027504e7b3c3d7b30fa4178906581da7/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_449a1bc2027504e7b3c3d7b30fa4178906581da7/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi124 with status 124: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json' During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_449a1bc2027504e7b3c3d7b30fa4178906581da7/teuthology/orchestra/daemon/state.py", line 139, in stop run.wait([self.proc], timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_449a1bc2027504e7b3c3d7b30fa4178906581da7/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_449a1bc2027504e7b3c3d7b30fa4178906581da7/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_449a1bc2027504e7b3c3d7b30fa4178906581da7/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi124 with status 5: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster cluster2 -i 0' 2023-08-26T18:15:46.228 INFO:tasks.ceph.cluster2.osd.0:Stopped 2023-08-26T18:15:46.229 DEBUG:tasks.ceph.cluster2.osd.1:waiting for process to exit 2023-08-26T18:15:46.229 INFO:teuthology.orchestra.run:waiting for 300 2023-08-26T18:15:46.230 DEBUG:teuthology.orchestra.run:got remote process result: 5 2023-08-26T18:15:46.231 ERROR:teuthology.orchestra.daemon.state:Error while waiting for process to exit Traceback (most recent call last): File "/home/teuthworker/src/github.com_ceph_ceph-c_e7eb9853fddaf661c68f207346131a7e00d8ed2f/qa/tasks/ceph.py", line 1940, in task yield File "/home/teuthworker/src/git.ceph.com_teuthology_449a1bc2027504e7b3c3d7b30fa4178906581da7/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ next(self.gen) File "/home/teuthworker/src/github.com_ceph_ceph-c_e7eb9853fddaf661c68f207346131a7e00d8ed2f/qa/tasks/rbd_mirror_thrash.py", line 216, in task raise RuntimeError('error during thrashing') RuntimeError: error during thrashing During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/github.com_ceph_ceph-c_e7eb9853fddaf661c68f207346131a7e00d8ed2f/qa/tasks/ceph.py", line 1469, in run_daemon yield File "/home/teuthworker/src/git.ceph.com_teuthology_449a1bc2027504e7b3c3d7b30fa4178906581da7/teuthology/contextutil.py", line 32, in nested yield vars File "/home/teuthworker/src/github.com_ceph_ceph-c_e7eb9853fddaf661c68f207346131a7e00d8ed2f/qa/tasks/ceph.py", line 1950, in task ctx.managers[config['cluster']].wait_for_clean() File "/home/teuthworker/src/github.com_ceph_ceph-c_e7eb9853fddaf661c68f207346131a7e00d8ed2f/qa/tasks/ceph_manager.py", line 2742, in wait_for_clean num_active_clean = self.get_num_active_clean() File "/home/teuthworker/src/github.com_ceph_ceph-c_e7eb9853fddaf661c68f207346131a7e00d8ed2f/qa/tasks/ceph_manager.py", line 2574, in get_num_active_clean pgs = self.get_pg_stats() File "/home/teuthworker/src/github.com_ceph_ceph-c_e7eb9853fddaf661c68f207346131a7e00d8ed2f/qa/tasks/ceph_manager.py", line 2340, in get_pg_stats out = self.raw_cluster_cmd('pg', 'dump', '--format=json') File "/home/teuthworker/src/github.com_ceph_ceph-c_e7eb9853fddaf661c68f207346131a7e00d8ed2f/qa/tasks/ceph_manager.py", line 1595, in raw_cluster_cmd return self.run_cluster_cmd(**kwargs).stdout.getvalue() File "/home/teuthworker/src/github.com_ceph_ceph-c_e7eb9853fddaf661c68f207346131a7e00d8ed2f/qa/tasks/ceph_manager.py", line 1586, in run_cluster_cmd return self.controller.run(**kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_449a1bc2027504e7b3c3d7b30fa4178906581da7/teuthology/orchestra/remote.py", line 522, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_449a1bc2027504e7b3c3d7b30fa4178906581da7/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_449a1bc2027504e7b3c3d7b30fa4178906581da7/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_449a1bc2027504e7b3c3d7b30fa4178906581da7/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi124 with status 124: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json' During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_449a1bc2027504e7b3c3d7b30fa4178906581da7/teuthology/orchestra/daemon/state.py", line 139, in stop run.wait([self.proc], timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_449a1bc2027504e7b3c3d7b30fa4178906581da7/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_449a1bc2027504e7b3c3d7b30fa4178906581da7/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_449a1bc2027504e7b3c3d7b30fa4178906581da7/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi124 with status 5: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster cluster2 -i 1' 2023-08-26T18:15:46.244 INFO:tasks.ceph.cluster2.osd.1:Stopped 2023-08-26T18:15:46.245 DEBUG:tasks.ceph.cluster2.osd.2:waiting for process to exit 2023-08-26T18:15:46.245 INFO:teuthology.orchestra.run:waiting for 300 2023-08-26T18:15:46.246 INFO:tasks.ceph.cluster2.osd.2:Stopped 2023-08-26T18:15:46.246 INFO:teuthology.misc:Shutting down mgr daemons... 2023-08-26T18:15:46.247 DEBUG:tasks.ceph.cluster2.mgr.x:waiting for process to exit 2023-08-26T18:15:46.247 INFO:teuthology.orchestra.run:waiting for 300 2023-08-26T18:15:46.248 INFO:tasks.ceph.cluster2.mgr.x:Stopped 2023-08-26T18:15:46.249 INFO:teuthology.misc:Shutting down mon daemons... 2023-08-26T18:15:46.249 DEBUG:tasks.ceph.cluster2.mon.a:waiting for process to exit 2023-08-26T18:15:46.250 INFO:teuthology.orchestra.run:waiting for 300 2023-08-26T18:15:46.250 INFO:tasks.ceph.cluster2.mon.a:Stopped 2023-08-26T18:15:46.251 INFO:tasks.ceph:Checking cluster log for badness... 2023-08-26T18:15:46.251 DEBUG:teuthology.orchestra.run.smithi124:> sudo egrep '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/cluster2.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v '\(OSD_SLOW_PING_TIME' | head -n 1 2023-08-26T18:15:46.287 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/cluster2-0 on ubuntu@smithi124.front.sepia.ceph.com 2023-08-26T18:15:46.288 DEBUG:teuthology.orchestra.run.smithi124:> sync && sudo umount -f /var/lib/ceph/osd/cluster2-0 2023-08-26T18:15:46.568 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/cluster2-1 on ubuntu@smithi124.front.sepia.ceph.com 2023-08-26T18:15:46.568 DEBUG:teuthology.orchestra.run.smithi124:> sync && sudo umount -f /var/lib/ceph/osd/cluster2-1 2023-08-26T18:15:46.764 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/cluster2-2 on ubuntu@smithi124.front.sepia.ceph.com 2023-08-26T18:15:46.764 DEBUG:teuthology.orchestra.run.smithi124:> sync && sudo umount -f /var/lib/ceph/osd/cluster2-2 2023-08-26T18:15:46.946 INFO:tasks.ceph:Archiving mon data... 2023-08-26T18:15:46.961 DEBUG:teuthology.misc:Transferring archived files from smithi124:/var/lib/ceph/mon/cluster2-a to /home/teuthworker/archive/yuriw-2023-08-25_17:40:36-rbd-main-distro-default-smithi/7380431/data/cluster2.mon.a.tgz 2023-08-26T18:15:46.962 DEBUG:teuthology.orchestra.run.smithi124:> mktemp 2023-08-26T18:15:46.980 INFO:teuthology.orchestra.run.smithi124.stdout:/tmp/tmp.wlBjGnOavD 2023-08-26T18:15:46.980 DEBUG:teuthology.orchestra.run.smithi124:> sudo tar cz -f - -C /var/lib/ceph/mon/cluster2-a -- . > /tmp/tmp.wlBjGnOavD 2023-08-26T18:15:47.168 DEBUG:teuthology.orchestra.run.smithi124:> sudo chmod 0666 /tmp/tmp.wlBjGnOavD 2023-08-26T18:15:47.259 DEBUG:teuthology.orchestra.remote:smithi124:/tmp/tmp.wlBjGnOavD is 557KB 2023-08-26T18:15:47.359 DEBUG:teuthology.orchestra.run.smithi124:> rm -fr /tmp/tmp.wlBjGnOavD 2023-08-26T18:15:47.377 INFO:tasks.ceph:Cleaning ceph cluster... 2023-08-26T18:15:47.378 DEBUG:teuthology.orchestra.run.smithi031:> sudo rm -rf -- /etc/ceph/cluster2.conf /etc/ceph/cluster2.keyring /home/ubuntu/cephtest/cluster2.data /home/ubuntu/cephtest/cluster2.monmap /home/ubuntu/cephtest/../*.pid 2023-08-26T18:15:47.383 DEBUG:teuthology.orchestra.run.smithi124:> sudo rm -rf -- /etc/ceph/cluster2.conf /etc/ceph/cluster2.keyring /home/ubuntu/cephtest/cluster2.data /home/ubuntu/cephtest/cluster2.monmap /home/ubuntu/cephtest/../*.pid 2023-08-26T18:15:47.443 ERROR:teuthology.run_tasks:Manager failed: ceph Traceback (most recent call last): File "/home/teuthworker/src/github.com_ceph_ceph-c_e7eb9853fddaf661c68f207346131a7e00d8ed2f/qa/tasks/ceph.py", line 1940, in task yield File "/home/teuthworker/src/git.ceph.com_teuthology_449a1bc2027504e7b3c3d7b30fa4178906581da7/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ next(self.gen) File "/home/teuthworker/src/github.com_ceph_ceph-c_e7eb9853fddaf661c68f207346131a7e00d8ed2f/qa/tasks/rbd_mirror_thrash.py", line 216, in task raise RuntimeError('error during thrashing') RuntimeError: error during thrashing During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_449a1bc2027504e7b3c3d7b30fa4178906581da7/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/github.com_ceph_ceph-c_e7eb9853fddaf661c68f207346131a7e00d8ed2f/qa/tasks/ceph.py", line 1950, in task ctx.managers[config['cluster']].wait_for_clean() File "/home/teuthworker/src/github.com_ceph_ceph-c_e7eb9853fddaf661c68f207346131a7e00d8ed2f/qa/tasks/ceph_manager.py", line 2742, in wait_for_clean num_active_clean = self.get_num_active_clean() File "/home/teuthworker/src/github.com_ceph_ceph-c_e7eb9853fddaf661c68f207346131a7e00d8ed2f/qa/tasks/ceph_manager.py", line 2574, in get_num_active_clean pgs = self.get_pg_stats() File "/home/teuthworker/src/github.com_ceph_ceph-c_e7eb9853fddaf661c68f207346131a7e00d8ed2f/qa/tasks/ceph_manager.py", line 2340, in get_pg_stats out = self.raw_cluster_cmd('pg', 'dump', '--format=json') File "/home/teuthworker/src/github.com_ceph_ceph-c_e7eb9853fddaf661c68f207346131a7e00d8ed2f/qa/tasks/ceph_manager.py", line 1595, in raw_cluster_cmd return self.run_cluster_cmd(**kwargs).stdout.getvalue() File "/home/teuthworker/src/github.com_ceph_ceph-c_e7eb9853fddaf661c68f207346131a7e00d8ed2f/qa/tasks/ceph_manager.py", line 1586, in run_cluster_cmd return self.controller.run(**kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_449a1bc2027504e7b3c3d7b30fa4178906581da7/teuthology/orchestra/remote.py", line 522, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_449a1bc2027504e7b3c3d7b30fa4178906581da7/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_449a1bc2027504e7b3c3d7b30fa4178906581da7/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_449a1bc2027504e7b3c3d7b30fa4178906581da7/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi124 with status 124: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json' 2023-08-26T18:15:47.468 DEBUG:teuthology.run_tasks:Unwinding manager ceph 2023-08-26T18:15:47.494 INFO:tasks.ceph.ceph_manager.cluster1:waiting for clean 2023-08-26T18:15:47.495 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 pg dump --format=json 2023-08-26T18:17:47.537 DEBUG:teuthology.orchestra.run:got remote process result: 124 2023-08-26T18:17:47.538 ERROR:teuthology.contextutil:Saw exception from nested tasks Traceback (most recent call last): File "/home/teuthworker/src/github.com_ceph_ceph-c_e7eb9853fddaf661c68f207346131a7e00d8ed2f/qa/tasks/ceph.py", line 1940, in task yield File "/home/teuthworker/src/git.ceph.com_teuthology_449a1bc2027504e7b3c3d7b30fa4178906581da7/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ next(self.gen) File "/home/teuthworker/src/github.com_ceph_ceph-c_e7eb9853fddaf661c68f207346131a7e00d8ed2f/qa/tasks/rbd_mirror_thrash.py", line 216, in task raise RuntimeError('error during thrashing') RuntimeError: error during thrashing During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_449a1bc2027504e7b3c3d7b30fa4178906581da7/teuthology/contextutil.py", line 32, in nested yield vars File "/home/teuthworker/src/github.com_ceph_ceph-c_e7eb9853fddaf661c68f207346131a7e00d8ed2f/qa/tasks/ceph.py", line 1950, in task ctx.managers[config['cluster']].wait_for_clean() File "/home/teuthworker/src/github.com_ceph_ceph-c_e7eb9853fddaf661c68f207346131a7e00d8ed2f/qa/tasks/ceph_manager.py", line 2742, in wait_for_clean num_active_clean = self.get_num_active_clean() File "/home/teuthworker/src/github.com_ceph_ceph-c_e7eb9853fddaf661c68f207346131a7e00d8ed2f/qa/tasks/ceph_manager.py", line 2574, in get_num_active_clean pgs = self.get_pg_stats() File "/home/teuthworker/src/github.com_ceph_ceph-c_e7eb9853fddaf661c68f207346131a7e00d8ed2f/qa/tasks/ceph_manager.py", line 2340, in get_pg_stats out = self.raw_cluster_cmd('pg', 'dump', '--format=json') File "/home/teuthworker/src/github.com_ceph_ceph-c_e7eb9853fddaf661c68f207346131a7e00d8ed2f/qa/tasks/ceph_manager.py", line 1595, in raw_cluster_cmd return self.run_cluster_cmd(**kwargs).stdout.getvalue() File "/home/teuthworker/src/github.com_ceph_ceph-c_e7eb9853fddaf661c68f207346131a7e00d8ed2f/qa/tasks/ceph_manager.py", line 1586, in run_cluster_cmd return self.controller.run(**kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_449a1bc2027504e7b3c3d7b30fa4178906581da7/teuthology/orchestra/remote.py", line 522, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_449a1bc2027504e7b3c3d7b30fa4178906581da7/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_449a1bc2027504e7b3c3d7b30fa4178906581da7/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_449a1bc2027504e7b3c3d7b30fa4178906581da7/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi031 with status 124: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 pg dump --format=json' 2023-08-26T18:17:47.553 INFO:teuthology.misc:Shutting down mds daemons... 2023-08-26T18:17:47.554 INFO:teuthology.misc:Shutting down osd daemons... 2023-08-26T18:17:47.554 DEBUG:tasks.ceph.cluster1.osd.0:waiting for process to exit 2023-08-26T18:17:47.555 INFO:teuthology.orchestra.run:waiting for 300 2023-08-26T18:17:47.559 INFO:tasks.ceph.cluster1.osd.0:Stopped 2023-08-26T18:17:47.560 DEBUG:tasks.ceph.cluster1.osd.1:waiting for process to exit 2023-08-26T18:17:47.560 INFO:teuthology.orchestra.run:waiting for 300 2023-08-26T18:17:47.561 INFO:tasks.ceph.cluster1.osd.1:Stopped 2023-08-26T18:17:47.561 DEBUG:tasks.ceph.cluster1.osd.2:waiting for process to exit 2023-08-26T18:17:47.562 INFO:teuthology.orchestra.run:waiting for 300 2023-08-26T18:17:47.562 INFO:tasks.ceph.cluster1.osd.2:Stopped 2023-08-26T18:17:47.563 INFO:teuthology.misc:Shutting down mgr daemons... 2023-08-26T18:17:47.563 DEBUG:tasks.ceph.cluster1.mgr.x:waiting for process to exit 2023-08-26T18:17:47.564 INFO:teuthology.orchestra.run:waiting for 300 2023-08-26T18:17:47.564 INFO:tasks.ceph.cluster1.mgr.x:Stopped 2023-08-26T18:17:47.565 INFO:teuthology.misc:Shutting down mon daemons... 2023-08-26T18:17:47.565 DEBUG:tasks.ceph.cluster1.mon.a:waiting for process to exit 2023-08-26T18:17:47.566 INFO:teuthology.orchestra.run:waiting for 300 2023-08-26T18:17:47.566 INFO:tasks.ceph.cluster1.mon.a:Stopped 2023-08-26T18:17:47.567 INFO:tasks.ceph:Checking cluster log for badness... 2023-08-26T18:17:47.568 DEBUG:teuthology.orchestra.run.smithi031:> sudo egrep '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/cluster1.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v '\(OSD_SLOW_PING_TIME' | head -n 1 2023-08-26T18:17:47.614 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/cluster1-0 on ubuntu@smithi031.front.sepia.ceph.com 2023-08-26T18:17:47.614 DEBUG:teuthology.orchestra.run.smithi031:> sync && sudo umount -f /var/lib/ceph/osd/cluster1-0 2023-08-26T18:17:47.768 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/cluster1-1 on ubuntu@smithi031.front.sepia.ceph.com 2023-08-26T18:17:47.769 DEBUG:teuthology.orchestra.run.smithi031:> sync && sudo umount -f /var/lib/ceph/osd/cluster1-1 2023-08-26T18:17:47.892 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/cluster1-2 on ubuntu@smithi031.front.sepia.ceph.com 2023-08-26T18:17:47.893 DEBUG:teuthology.orchestra.run.smithi031:> sync && sudo umount -f /var/lib/ceph/osd/cluster1-2 2023-08-26T18:17:47.990 INFO:tasks.ceph:Archiving mon data... 2023-08-26T18:17:47.991 DEBUG:teuthology.misc:Transferring archived files from smithi031:/var/lib/ceph/mon/cluster1-a to /home/teuthworker/archive/yuriw-2023-08-25_17:40:36-rbd-main-distro-default-smithi/7380431/data/cluster1.mon.a.tgz 2023-08-26T18:17:47.991 DEBUG:teuthology.orchestra.run.smithi031:> mktemp 2023-08-26T18:17:48.009 INFO:teuthology.orchestra.run.smithi031.stdout:/tmp/tmp.xotVZpOd4Z 2023-08-26T18:17:48.010 DEBUG:teuthology.orchestra.run.smithi031:> sudo tar cz -f - -C /var/lib/ceph/mon/cluster1-a -- . > /tmp/tmp.xotVZpOd4Z 2023-08-26T18:17:48.162 DEBUG:teuthology.orchestra.run.smithi031:> sudo chmod 0666 /tmp/tmp.xotVZpOd4Z 2023-08-26T18:17:48.251 DEBUG:teuthology.orchestra.remote:smithi031:/tmp/tmp.xotVZpOd4Z is 354KB 2023-08-26T18:17:48.355 DEBUG:teuthology.orchestra.run.smithi031:> rm -fr /tmp/tmp.xotVZpOd4Z 2023-08-26T18:17:48.373 INFO:tasks.ceph:Cleaning ceph cluster... 2023-08-26T18:17:48.374 DEBUG:teuthology.orchestra.run.smithi031:> sudo rm -rf -- /etc/ceph/cluster1.conf /etc/ceph/cluster1.keyring /home/ubuntu/cephtest/cluster1.data /home/ubuntu/cephtest/cluster1.monmap /home/ubuntu/cephtest/../*.pid 2023-08-26T18:17:48.415 DEBUG:teuthology.orchestra.run.smithi124:> sudo rm -rf -- /etc/ceph/cluster1.conf /etc/ceph/cluster1.keyring /home/ubuntu/cephtest/cluster1.data /home/ubuntu/cephtest/cluster1.monmap /home/ubuntu/cephtest/../*.pid 2023-08-26T18:17:48.450 INFO:tasks.ceph:Checking for errors in any valgrind logs... 2023-08-26T18:17:48.451 DEBUG:teuthology.orchestra.run.smithi031:> sudo zgrep '' /var/log/ceph/valgrind/* /dev/null | sort | uniq 2023-08-26T18:17:48.484 DEBUG:teuthology.orchestra.run.smithi124:> sudo zgrep '' /var/log/ceph/valgrind/* /dev/null | sort | uniq 2023-08-26T18:17:48.522 INFO:teuthology.orchestra.run.smithi031.stderr:gzip: /var/log/ceph/valgrind/*.gz: No such file or directory 2023-08-26T18:17:48.622 INFO:teuthology.orchestra.run.smithi124.stderr:gzip: /var/log/ceph/valgrind/*.gz: No such file or directory 2023-08-26T18:17:48.629 INFO:tasks.ceph:Archiving crash dumps... 2023-08-26T18:17:48.635 DEBUG:teuthology.misc:Transferring archived files from smithi031:/var/lib/ceph/crash to /home/teuthworker/archive/yuriw-2023-08-25_17:40:36-rbd-main-distro-default-smithi/7380431/remote/smithi031/crash 2023-08-26T18:17:48.641 DEBUG:teuthology.orchestra.run.smithi031:> sudo tar c -f - -C /var/lib/ceph/crash -- . 2023-08-26T18:17:48.673 DEBUG:teuthology.misc:Transferring archived files from smithi124:/var/lib/ceph/crash to /home/teuthworker/archive/yuriw-2023-08-25_17:40:36-rbd-main-distro-default-smithi/7380431/remote/smithi124/crash 2023-08-26T18:17:48.675 DEBUG:teuthology.orchestra.run.smithi124:> sudo tar c -f - -C /var/lib/ceph/crash -- . 2023-08-26T18:17:48.821 INFO:tasks.ceph:Compressing logs... 2023-08-26T18:17:48.822 DEBUG:teuthology.orchestra.run.smithi031:> time sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2023-08-26T18:17:48.825 DEBUG:teuthology.orchestra.run.smithi124:> time sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2023-08-26T18:17:48.851 INFO:teuthology.orchestra.run.smithi031.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.99234.log 2023-08-26T18:17:48.852 INFO:teuthology.orchestra.run.smithi031.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.104665.log 2023-08-26T18:17:48.852 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/cluster1-client.admin.99234.log: gzip -5 --verbose -- /var/log/ceph/cluster1-osd.0.log 2023-08-26T18:17:48.852 INFO:teuthology.orchestra.run.smithi031.stderr: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.99234.log.gz 2023-08-26T18:17:48.852 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/cluster1-client.admin.104665.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.104665.log.gz 2023-08-26T18:17:48.853 INFO:teuthology.orchestra.run.smithi031.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.104068.log 2023-08-26T18:17:48.853 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/cluster1-osd.0.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.104758.log 2023-08-26T18:17:48.853 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/cluster1-client.admin.104068.log: 0.0%gzip -- replaced with /var/log/ceph/cluster1-client.admin.104068.log.gz 2023-08-26T18:17:48.854 INFO:teuthology.orchestra.run.smithi031.stderr:-5 --verbose -- /var/log/ceph/cluster1-client.admin.104252.log 2023-08-26T18:17:48.863 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/cluster1-client.admin.104758.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.104482.log 2023-08-26T18:17:48.864 INFO:teuthology.orchestra.run.smithi031.stderr: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.104758.log.gz 2023-08-26T18:17:48.864 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/cluster1-client.admin.104252.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.99352.log 2023-08-26T18:17:48.865 INFO:teuthology.orchestra.run.smithi031.stderr: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.104252.log.gz 2023-08-26T18:17:48.865 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/cluster1-client.admin.104482.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.104620.log 2023-08-26T18:17:48.865 INFO:teuthology.orchestra.run.smithi031.stderr: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.104482.log.gz 2023-08-26T18:17:48.865 INFO:teuthology.orchestra.run.smithi031.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.104390.log 2023-08-26T18:17:48.866 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/cluster1-client.admin.99352.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.99352.log.gz 2023-08-26T18:17:48.866 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/cluster1-client.admin.104620.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.104160.log 2023-08-26T18:17:48.866 INFO:teuthology.orchestra.run.smithi031.stderr: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.104620.log.gz 2023-08-26T18:17:48.866 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/cluster1-client.admin.104390.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.109564.log 2023-08-26T18:17:48.867 INFO:teuthology.orchestra.run.smithi031.stderr: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.104390.log.gz 2023-08-26T18:17:48.867 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/cluster1-client.admin.104160.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.104114.log 2023-08-26T18:17:48.868 INFO:teuthology.orchestra.run.smithi031.stderr: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.104160.log.gz 2023-08-26T18:17:48.868 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/cluster1-client.admin.109564.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.103976.log 2023-08-26T18:17:48.868 INFO:teuthology.orchestra.run.smithi031.stderr: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.109564.log.gz 2023-08-26T18:17:48.868 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/cluster1-client.admin.104114.log: gzip -5 --verbose -- /var/log/ceph/cluster2-mgr.x.log 2023-08-26T18:17:48.869 INFO:teuthology.orchestra.run.smithi031.stderr: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.104114.log.gz 2023-08-26T18:17:48.869 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/cluster1-client.admin.103976.log: gzip 0.0%-5 -- replaced with /var/log/ceph/cluster1-client.admin.103976.log.gz--verbose 2023-08-26T18:17:48.869 INFO:teuthology.orchestra.run.smithi031.stderr:-- /var/log/ceph/ceph.tmp-client.admin.94307.log 2023-08-26T18:17:48.870 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/cluster2-mgr.x.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.99188.log 2023-08-26T18:17:48.870 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/ceph.tmp-client.admin.94307.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.104250.log 2023-08-26T18:17:48.870 INFO:teuthology.orchestra.run.smithi031.stderr: 0.0% -- replaced with /var/log/ceph/ceph.tmp-client.admin.94307.log.gz 2023-08-26T18:17:48.870 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/cluster1-client.admin.99188.log: gzip 0.0%-5 -- replaced with /var/log/ceph/cluster1-client.admin.99188.log.gz--verbose 2023-08-26T18:17:48.871 INFO:teuthology.orchestra.run.smithi031.stderr:-- /var/log/ceph/cluster1-client.admin.104528.log 2023-08-26T18:17:48.871 INFO:teuthology.orchestra.run.smithi031.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.103823.log 2023-08-26T18:17:48.871 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/cluster1-client.admin.104250.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.104250.log.gz 2023-08-26T18:17:48.872 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/cluster1-client.admin.104528.log: gzip -5 --verbose -- /var/log/ceph/cluster1-osd.1.log 2023-08-26T18:17:48.872 INFO:teuthology.orchestra.run.smithi031.stderr: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.104528.log.gz 2023-08-26T18:17:48.873 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/cluster1-client.admin.103823.log: gzip -5 --verbose -- /var/log/ceph/cluster1-mon.a.log 2023-08-26T18:17:48.873 INFO:teuthology.orchestra.run.smithi031.stderr: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.103823.log.gz 2023-08-26T18:17:48.873 INFO:teuthology.orchestra.run.smithi031.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.104022.log 2023-08-26T18:17:48.874 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/cluster1-osd.1.log: /var/log/ceph/cluster1-mon.a.log: /var/log/ceph/cluster1-client.admin.104022.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.104022.log.gz 2023-08-26T18:17:48.874 INFO:teuthology.orchestra.run.smithi031.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.104850.log 2023-08-26T18:17:48.874 INFO:teuthology.orchestra.run.smithi031.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.98968.log 2023-08-26T18:17:48.877 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/cluster1-client.admin.104850.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.104850.log.gz 2023-08-26T18:17:48.877 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/cluster1-client.admin.98968.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.98968.log.gz 2023-08-26T18:17:48.878 INFO:teuthology.orchestra.run.smithi031.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.99141.log 2023-08-26T18:17:48.878 INFO:teuthology.orchestra.run.smithi031.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.104804.log 2023-08-26T18:17:48.878 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/cluster1-client.admin.99141.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.99141.log.gz 2023-08-26T18:17:48.879 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/cluster1-client.admin.104804.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.104804.log.gz 2023-08-26T18:17:48.879 INFO:teuthology.orchestra.run.smithi031.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1.log 2023-08-26T18:17:48.879 INFO:teuthology.orchestra.run.smithi031.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.103913.log 2023-08-26T18:17:48.879 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/cluster1.log: /var/log/ceph/cluster1-client.admin.103913.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.103913.log.gz 2023-08-26T18:17:48.880 INFO:teuthology.orchestra.run.smithi031.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1.audit.log 2023-08-26T18:17:48.880 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/cluster1.audit.log: gzip -5 --verbose -- /var/log/ceph/cluster1-osd.2.log 2023-08-26T18:17:48.880 INFO:teuthology.orchestra.run.smithi031.stderr: 93.1% -- replaced with /var/log/ceph/cluster1.audit.log.gz 2023-08-26T18:17:48.881 INFO:teuthology.orchestra.run.smithi031.stderr: 88.5% -- replaced with /var/log/ceph/cluster1.log.gz 2023-08-26T18:17:48.881 INFO:teuthology.orchestra.run.smithi031.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.104527.log 2023-08-26T18:17:48.882 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/cluster1-osd.2.log: /var/log/ceph/cluster1-client.admin.104527.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.104527.log.gz 2023-08-26T18:17:48.882 INFO:teuthology.orchestra.run.smithi031.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.104251.log 2023-08-26T18:17:48.882 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/cluster1-client.admin.104251.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.104251.log.gz 2023-08-26T18:17:48.883 INFO:teuthology.orchestra.run.smithi031.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.104388.log 2023-08-26T18:17:48.883 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/cluster1-client.admin.104388.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.104388.log.gz 2023-08-26T18:17:48.883 INFO:teuthology.orchestra.run.smithi031.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.104389.log 2023-08-26T18:17:48.883 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/cluster1-client.admin.104389.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.104389.log.gz 2023-08-26T18:17:48.884 INFO:teuthology.orchestra.run.smithi031.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.99422.log 2023-08-26T18:17:48.884 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/cluster1-client.admin.99422.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.99422.log.gz 2023-08-26T18:17:48.884 INFO:teuthology.orchestra.run.smithi031.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.104666.log 2023-08-26T18:17:48.884 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/cluster1-client.admin.104666.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.104666.log.gz 2023-08-26T18:17:48.885 INFO:teuthology.orchestra.run.smithi031.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-mgr.x.log 2023-08-26T18:17:48.907 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.133859.log 2023-08-26T18:17:48.908 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.135878.log 2023-08-26T18:17:48.913 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster2-client.mirror.133859.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.115246.log 2023-08-26T18:17:48.914 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster2-client.mirror.135878.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.134802.log 2023-08-26T18:17:48.914 INFO:teuthology.orchestra.run.smithi124.stderr: 88.4% -- replaced with /var/log/ceph/cluster2-client.mirror.133859.log.gz 2023-08-26T18:17:48.914 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster2-client.mirror.115246.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.113229.log 2023-08-26T18:17:48.914 INFO:teuthology.orchestra.run.smithi124.stderr: 87.4%/var/log/ceph/cluster2-client.mirror.134802.log: -- replaced with /var/log/ceph/cluster2-client.mirror.115246.log.gz 2023-08-26T18:17:48.915 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.132967.log 2023-08-26T18:17:48.915 INFO:teuthology.orchestra.run.smithi124.stderr: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.134802.log.gz 2023-08-26T18:17:48.915 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster2-client.mirror.113229.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.110398.log 2023-08-26T18:17:48.916 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster1-client.mirror.132967.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.134065.log 2023-08-26T18:17:48.916 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster2-client.mirror.110398.log: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.132967.log.gz 2023-08-26T18:17:48.916 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.138815.log 2023-08-26T18:17:48.916 INFO:teuthology.orchestra.run.smithi124.stderr: 90.5% -- replaced with /var/log/ceph/cluster2-client.mirror.135878.log.gz 2023-08-26T18:17:48.917 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.132655.log 2023-08-26T18:17:48.917 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster2-client.mirror.134065.log: /var/log/ceph/cluster1-client.mirror.138815.log: gzip -5 --verbose -- /var/log/ceph/cluster2-osd.0.log 2023-08-26T18:17:48.917 INFO:teuthology.orchestra.run.smithi124.stderr: 86.8% -- replaced with /var/log/ceph/cluster1-client.mirror.138815.log.gz 2023-08-26T18:17:48.917 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster2-client.mirror.132655.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.119356.log 2023-08-26T18:17:48.918 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster2-osd.0.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.115413.log 2023-08-26T18:17:48.918 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster2-client.mirror.119356.log: 89.8% -- replaced with /var/log/ceph/cluster2-client.mirror.132655.log.gz 2023-08-26T18:17:48.918 INFO:teuthology.orchestra.run.smithi124.stderr: 79.5% -- replaced with /var/log/ceph/cluster2-client.mirror.119356.log.gz 2023-08-26T18:17:48.919 INFO:teuthology.orchestra.run.smithi124.stderr: 90.4% -- replaced with /var/log/ceph/cluster2-client.mirror.134065.log.gz 2023-08-26T18:17:48.919 INFO:teuthology.orchestra.run.smithi124.stderr: 93.4% -- replaced with /var/log/ceph/cluster2-client.mirror.113229.log.gz 2023-08-26T18:17:48.927 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.109298.log 2023-08-26T18:17:48.928 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster2-client.mirror.115413.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.127737.log 2023-08-26T18:17:48.928 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster2-client.mirror.109298.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.137420.log 2023-08-26T18:17:48.928 INFO:teuthology.orchestra.run.smithi124.stderr: 89.7% -- replaced with /var/log/ceph/cluster2-client.mirror.115413.log.gz 2023-08-26T18:17:48.929 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster2-client.mirror.127737.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.114657.log 2023-08-26T18:17:48.929 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster1-client.mirror.137420.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.115159.log 2023-08-26T18:17:48.929 INFO:teuthology.orchestra.run.smithi124.stderr: 89.8% -- replaced with /var/log/ceph/cluster2-client.mirror.127737.log.gz 2023-08-26T18:17:48.930 INFO:teuthology.orchestra.run.smithi124.stderr: 86.7% -- replaced with /var/log/ceph/cluster1-client.mirror.137420.log.gz 2023-08-26T18:17:48.930 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster2-client.mirror.114657.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.132792.log 2023-08-26T18:17:48.930 INFO:teuthology.orchestra.run.smithi124.stderr: 88.1% -- replaced with /var/log/ceph/cluster2-client.mirror.114657.log.gz 2023-08-26T18:17:48.930 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster2-client.mirror.115159.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.109255.log 2023-08-26T18:17:48.931 INFO:teuthology.orchestra.run.smithi124.stderr: 87.3% -- replaced with /var/log/ceph/cluster2-client.mirror.115159.log.gz 2023-08-26T18:17:48.931 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster2-client.mirror.132792.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.130258.log 2023-08-26T18:17:48.931 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster2-client.mirror.109255.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.111524.log 2023-08-26T18:17:48.931 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster1-client.mirror.130258.log: 87.2% -- replaced with /var/log/ceph/cluster1-client.mirror.130258.log.gz 2023-08-26T18:17:48.932 INFO:teuthology.orchestra.run.smithi124.stderr: 92.4% -- replaced with /var/log/ceph/cluster2-client.mirror.132792.log.gz 2023-08-26T18:17:48.932 INFO:teuthology.orchestra.run.smithi124.stderr: 93.3% -- replaced with /var/log/ceph/cluster2-client.mirror.109298.log.gz 2023-08-26T18:17:48.932 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.138883.log 2023-08-26T18:17:48.932 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster2-client.mirror.111524.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.111502.log 2023-08-26T18:17:48.933 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster1-client.mirror.138883.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.118988.log 2023-08-26T18:17:48.933 INFO:teuthology.orchestra.run.smithi124.stderr: 86.7% -- replaced with /var/log/ceph/cluster1-client.mirror.138883.log.gz 2023-08-26T18:17:48.933 INFO:teuthology.orchestra.run.smithi124.stderr: 89.8% -- replaced with /var/log/ceph/cluster2-client.mirror.111524.log.gz 2023-08-26T18:17:48.934 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster1-client.mirror.111502.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.136189.log 2023-08-26T18:17:48.934 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster2-client.mirror.118988.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.1.107760.log 2023-08-26T18:17:48.934 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster2-client.mirror.136189.log: 87.4% -- replaced with /var/log/ceph/cluster2-client.mirror.118988.log.gz 2023-08-26T18:17:48.934 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.132043.log 2023-08-26T18:17:48.935 INFO:teuthology.orchestra.run.smithi124.stderr: 88.1% -- replaced with /var/log/ceph/cluster2-client.mirror.136189.log.gz 2023-08-26T18:17:48.935 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster1-client.mirror.1.107760.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.124291.log 2023-08-26T18:17:48.935 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster1-client.mirror.132043.log: 87.8% -- replaced with /var/log/ceph/cluster1-client.mirror.132043.log.gz 2023-08-26T18:17:48.935 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster1-client.mirror.124291.log: 87.2% -- replaced with /var/log/ceph/cluster1-client.mirror.124291.log.gz 2023-08-26T18:17:48.936 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.131202.log 2023-08-26T18:17:48.936 INFO:teuthology.orchestra.run.smithi124.stderr: 93.3% -- replaced with /var/log/ceph/cluster2-client.mirror.109255.log.gz 2023-08-26T18:17:48.936 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.120429.log 2023-08-26T18:17:48.936 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster1-client.mirror.131202.log: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.131202.log.gz 2023-08-26T18:17:48.937 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster1-client.mirror.120429.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.120429.log.gz 2023-08-26T18:17:48.937 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.129703.log 2023-08-26T18:17:48.937 INFO:teuthology.orchestra.run.smithi124.stderr: 93.3% -- replaced with /var/log/ceph/cluster1-client.mirror.111502.log.gz 2023-08-26T18:17:48.937 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.129052.log 2023-08-26T18:17:48.938 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster1-client.mirror.129703.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.136010.log 2023-08-26T18:17:48.938 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster2-client.mirror.129052.log: 87.6% -- replaced with /var/log/ceph/cluster1-client.mirror.129703.log.gz 2023-08-26T18:17:48.938 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.114537.log 2023-08-26T18:17:48.939 INFO:teuthology.orchestra.run.smithi124.stderr: 93.4% -- replaced with /var/log/ceph/cluster1-client.mirror.1.107760.log.gz 2023-08-26T18:17:48.939 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.137375.log 2023-08-26T18:17:48.939 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster2-client.mirror.136010.log: /var/log/ceph/cluster2-client.mirror.114537.log: 89.7% -- replaced with /var/log/ceph/cluster2-client.mirror.129052.log.gz 2023-08-26T18:17:48.940 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.121639.log 2023-08-26T18:17:48.940 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster1-client.mirror.137375.log: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.114537.log.gz 2023-08-26T18:17:48.940 INFO:teuthology.orchestra.run.smithi124.stderr: 86.8% -- replaced with /var/log/ceph/cluster1-client.mirror.137375.log.gz 2023-08-26T18:17:48.940 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster1-client.mirror.121639.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.122299.log 2023-08-26T18:17:48.941 INFO:teuthology.orchestra.run.smithi124.stderr: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.121639.log.gz 2023-08-26T18:17:48.941 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.132991.log 2023-08-26T18:17:48.941 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.126464.log 2023-08-26T18:17:48.951 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster2-client.mirror.122299.log: /var/log/ceph/cluster1-client.mirror.132991.log: gzip 90.4%-5 -- replaced with /var/log/ceph/cluster2-client.mirror.136010.log.gz--verbose 2023-08-26T18:17:48.951 INFO:teuthology.orchestra.run.smithi124.stderr:-- /var/log/ceph/cluster1-client.mirror.108593.log 2023-08-26T18:17:48.951 INFO:teuthology.orchestra.run.smithi124.stderr: 87.3% -- replaced with /var/log/ceph/cluster2-client.mirror.122299.log.gz 2023-08-26T18:17:48.952 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster1-client.mirror.126464.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.118842.log 87.7% 2023-08-26T18:17:48.952 INFO:teuthology.orchestra.run.smithi124.stderr: -- replaced with /var/log/ceph/cluster1-client.mirror.132991.log.gz 2023-08-26T18:17:48.952 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster1-client.mirror.108593.log: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.126464.log.gz 2023-08-26T18:17:48.953 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.119205.log 2023-08-26T18:17:48.953 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster2-client.mirror.118842.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.112250.log 2023-08-26T18:17:48.953 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster2-client.mirror.119205.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.129191.log 2023-08-26T18:17:48.953 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster2-client.mirror.112250.log: /var/log/ceph/cluster1-client.mirror.129191.log: 89.1% 89.6% -- replaced with /var/log/ceph/cluster2-client.mirror.112250.log.gz 2023-08-26T18:17:48.954 INFO:teuthology.orchestra.run.smithi124.stderr: -- replaced with /var/log/ceph/cluster2-client.mirror.119205.log.gz 2023-08-26T18:17:48.954 INFO:teuthology.orchestra.run.smithi124.stderr: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.129191.log.gz 2023-08-26T18:17:48.954 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.123918.log 2023-08-26T18:17:48.954 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.140444.log 2023-08-26T18:17:48.955 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster2-client.mirror.123918.log: /var/log/ceph/cluster2-client.mirror.140444.log: 89.8% -- replaced with /var/log/ceph/cluster2-client.mirror.123918.log.gz 2023-08-26T18:17:48.955 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.121220.log 2023-08-26T18:17:48.955 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.122647.log 2023-08-26T18:17:48.956 INFO:teuthology.orchestra.run.smithi124.stderr: 91.5% -- replaced with /var/log/ceph/cluster2-client.mirror.140444.log.gz 2023-08-26T18:17:48.956 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster2-client.mirror.121220.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.134470.log 2023-08-26T18:17:48.956 INFO:teuthology.orchestra.run.smithi124.stderr: 93.4% -- replaced with /var/log/ceph/cluster1-client.mirror.108593.log.gz 2023-08-26T18:17:48.956 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster1-client.mirror.122647.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.133735.log 2023-08-26T18:17:48.957 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster2-client.mirror.134470.log: 87.6% -- replaced with /var/log/ceph/cluster1-client.mirror.122647.log.gz 2023-08-26T18:17:48.957 INFO:teuthology.orchestra.run.smithi124.stderr: 88.0% -- replaced with /var/log/ceph/cluster2-client.mirror.134470.log.gz 2023-08-26T18:17:48.957 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster2-client.mirror.133735.log: 92.5% -- replaced with /var/log/ceph/cluster2-client.mirror.121220.log.gz 2023-08-26T18:17:48.957 INFO:teuthology.orchestra.run.smithi124.stderr: 88.1% -- replaced with /var/log/ceph/cluster2-client.mirror.133735.log.gz 2023-08-26T18:17:48.958 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.125579.log 2023-08-26T18:17:48.958 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.125464.log 2023-08-26T18:17:48.958 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster2-client.mirror.125579.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.106660.log 2023-08-26T18:17:48.959 INFO:teuthology.orchestra.run.smithi124.stderr: 87.5% -- replaced with /var/log/ceph/cluster2-client.mirror.125579.log.gz 2023-08-26T18:17:48.959 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster2-client.mirror.125464.log: /var/log/ceph/cluster1-client.mirror.106660.log: 2.3% -- replaced with /var/log/ceph/cluster1-client.mirror.106660.log.gz 2023-08-26T18:17:48.960 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.106935.log 2023-08-26T18:17:48.960 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.130638.log 2023-08-26T18:17:48.960 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster2-client.mirror.106935.log: 54.2% -- replaced with /var/log/ceph/cluster2-client.mirror.106935.log.gz 2023-08-26T18:17:48.960 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster2-client.mirror.130638.log: 92.6% -- replaced with /var/log/ceph/cluster2-client.mirror.125464.log.gz 2023-08-26T18:17:48.961 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.117290.log 2023-08-26T18:17:48.961 INFO:teuthology.orchestra.run.smithi124.stderr: 92.5% -- replaced with /var/log/ceph/cluster2-client.mirror.130638.log.gz 2023-08-26T18:17:48.961 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.112767.log 2023-08-26T18:17:48.961 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.127864.log 2023-08-26T18:17:48.962 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster1-client.mirror.117290.log: /var/log/ceph/cluster1-client.admin.112767.log: 88.2% -- replaced with /var/log/ceph/cluster1-client.admin.112767.log.gz 2023-08-26T18:17:48.962 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster1-client.mirror.127864.log: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.127864.log.gz 2023-08-26T18:17:48.962 INFO:teuthology.orchestra.run.smithi124.stderr: 95.0% -- replaced with /var/log/ceph/cluster1-client.mirror.117290.log.gz 2023-08-26T18:17:48.962 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.121240.log 2023-08-26T18:17:48.963 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.121895.log 2023-08-26T18:17:48.963 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster1-client.mirror.121240.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.134045.log 2023-08-26T18:17:48.963 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster1-client.mirror.121895.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.110130.log 2023-08-26T18:17:48.964 INFO:teuthology.orchestra.run.smithi124.stderr: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.121895.log.gz 2023-08-26T18:17:48.964 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster2-client.mirror.134045.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.114597.log 2023-08-26T18:17:48.965 INFO:teuthology.orchestra.run.smithi124.stderr: 88.1% -- replaced with /var/log/ceph/cluster2-client.mirror.134045.log.gz 2023-08-26T18:17:48.965 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster2-client.mirror.110130.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.121728.log 2023-08-26T18:17:48.965 INFO:teuthology.orchestra.run.smithi124.stderr: 92.3% -- replaced with /var/log/ceph/cluster1-client.mirror.121240.log.gz 2023-08-26T18:17:48.965 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster2-client.mirror.114597.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.124621.log 2023-08-26T18:17:48.966 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster1-client.mirror.121728.log: 90.0% 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.110130.log.gz -- replaced with /var/log/ceph/cluster2-client.mirror.114597.log.gz 2023-08-26T18:17:48.966 INFO:teuthology.orchestra.run.smithi124.stderr: 2023-08-26T18:17:48.966 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster2-client.mirror.124621.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.109041.log 2023-08-26T18:17:48.966 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.119126.log 2023-08-26T18:17:48.967 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster2-client.mirror.109041.log: /var/log/ceph/cluster2-client.mirror.119126.log: 92.3% -- replaced with /var/log/ceph/cluster1-client.mirror.121728.log.gz 2023-08-26T18:17:48.967 INFO:teuthology.orchestra.run.smithi124.stderr: 89.7% -- replaced with /var/log/ceph/cluster2-client.mirror.109041.log.gz 2023-08-26T18:17:48.967 INFO:teuthology.orchestra.run.smithi124.stderr: 89.7% -- replaced with /var/log/ceph/cluster2-client.mirror.119126.log.gz 2023-08-26T18:17:48.967 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.122960.log 2023-08-26T18:17:48.968 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.116367.log 2023-08-26T18:17:48.968 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster2-client.mirror.122960.log: /var/log/ceph/cluster2-client.mirror.116367.log: 87.4% -- replaced with /var/log/ceph/cluster2-client.mirror.116367.log.gz 2023-08-26T18:17:48.968 INFO:teuthology.orchestra.run.smithi124.stderr: 89.6% -- replaced with /var/log/ceph/cluster2-client.mirror.122960.log.gz 2023-08-26T18:17:48.969 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.108049.log 2023-08-26T18:17:48.969 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.135153.log 2023-08-26T18:17:48.969 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster1-client.mirror.108049.log: /var/log/ceph/cluster2-client.mirror.135153.log: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.108049.log.gz 2023-08-26T18:17:48.969 INFO:teuthology.orchestra.run.smithi124.stderr: 88.0% -- replaced with /var/log/ceph/cluster2-client.mirror.135153.log.gz 2023-08-26T18:17:48.970 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.108644.log 2023-08-26T18:17:48.970 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.134115.log 2023-08-26T18:17:48.970 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster1-client.mirror.108644.log: /var/log/ceph/cluster2-client.mirror.134115.log: 88.1% -- replaced with /var/log/ceph/cluster2-client.mirror.134115.log.gz 2023-08-26T18:17:48.970 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.134553.log 2023-08-26T18:17:48.971 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.121030.log 2023-08-26T18:17:48.975 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster2-client.mirror.134553.log: /var/log/ceph/cluster2-client.mirror.121030.log: 88.1% -- replaced with /var/log/ceph/cluster2-client.mirror.134553.log.gz 2023-08-26T18:17:48.975 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.140902.log 2023-08-26T18:17:48.976 INFO:teuthology.orchestra.run.smithi124.stderr: 89.7% -- replaced with /var/log/ceph/cluster2-client.mirror.121030.log.gz 2023-08-26T18:17:48.976 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster1-client.mirror.140902.log: 86.8% -- replaced with /var/log/ceph/cluster1-client.mirror.140902.log.gz 2023-08-26T18:17:48.977 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.118259.log 2023-08-26T18:17:48.977 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.134822.log 2023-08-26T18:17:48.977 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster2-client.mirror.118259.log: 89.6% -- replaced with /var/log/ceph/cluster2-client.mirror.118259.log.gz 2023-08-26T18:17:48.978 INFO:teuthology.orchestra.run.smithi124.stderr: 93.4% -- replaced with /var/log/ceph/cluster1-client.mirror.108644.log.gz 2023-08-26T18:17:48.978 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.133552.log 2023-08-26T18:17:48.978 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster2-client.mirror.134822.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.138995.log 2023-08-26T18:17:48.979 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster1-client.mirror.133552.log: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.133552.log.gz 2023-08-26T18:17:48.979 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.140684.log 2023-08-26T18:17:48.979 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster1-client.mirror.138995.log: 86.8% -- replaced with /var/log/ceph/cluster1-client.mirror.138995.log.gz 2023-08-26T18:17:48.979 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster2-client.mirror.140684.log: 88.1% -- replaced with /var/log/ceph/cluster2-client.mirror.140684.log.gz 2023-08-26T18:17:48.980 INFO:teuthology.orchestra.run.smithi124.stderr: 90.4% -- replaced with /var/log/ceph/cluster2-client.mirror.134822.log.gz 2023-08-26T18:17:48.980 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.119247.log 2023-08-26T18:17:48.980 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.115965.log 2023-08-26T18:17:48.980 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster2-client.mirror.119247.log: /var/log/ceph/cluster2-client.mirror.115965.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.141943.log 2023-08-26T18:17:48.981 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster1-client.mirror.141943.log: 87.2% -- replaced with /var/log/ceph/cluster1-client.mirror.141943.log.gz 2023-08-26T18:17:48.981 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.125101.log 2023-08-26T18:17:48.981 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster2-client.mirror.125101.log: 87.3% -- replaced with /var/log/ceph/cluster2-client.mirror.125101.log.gz 2023-08-26T18:17:48.982 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.112230.log 2023-08-26T18:17:48.982 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster2-client.mirror.112230.log: 90.0% -- replaced with /var/log/ceph/cluster2-client.mirror.112230.log.gz 2023-08-26T18:17:48.982 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.124027.log 2023-08-26T18:17:48.982 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster1-client.mirror.124027.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.124027.log.gz 2023-08-26T18:17:48.983 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.120943.log 2023-08-26T18:17:48.983 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster1-client.mirror.120943.log: 87.2% -- replaced with /var/log/ceph/cluster1-client.mirror.120943.log.gz 2023-08-26T18:17:48.983 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.137825.log 2023-08-26T18:17:48.983 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster1-client.mirror.137825.log: 86.8% -- replaced with /var/log/ceph/cluster1-client.mirror.137825.log.gz 2023-08-26T18:17:48.984 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.117379.log 2023-08-26T18:17:48.984 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.131021.log 2023-08-26T18:17:48.987 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.107691.log 2023-08-26T18:17:48.988 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster2-client.mirror.131021.log: /var/log/ceph/cluster1-client.admin.107691.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.admin.107691.log.gz 2023-08-26T18:17:48.988 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.115070.log 2023-08-26T18:17:48.988 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster2-client.mirror.117379.log: 92.4% -- replaced with /var/log/ceph/cluster2-client.mirror.131021.log.gz 2023-08-26T18:17:48.988 INFO:teuthology.orchestra.run.smithi124.stderr: 79.4% -- replaced with /var/log/ceph/cluster2-client.mirror.117379.log.gz 2023-08-26T18:17:48.989 INFO:teuthology.orchestra.run.smithi124.stderr: 93.4% -- replaced with /var/log/ceph/cluster2-client.mirror.124621.log.gz 2023-08-26T18:17:48.989 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster1-client.mirror.115070.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.120147.log 2023-08-26T18:17:48.989 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster2-client.mirror.120147.log: 95.2% -- replaced with /var/log/ceph/cluster1-client.mirror.115070.log.gz 2023-08-26T18:17:48.989 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.118242.log 2023-08-26T18:17:48.990 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster2-client.mirror.118242.log: 79.7% -- replaced with /var/log/ceph/cluster2-client.mirror.118242.log.gz 2023-08-26T18:17:48.990 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.129766.log 2023-08-26T18:17:48.990 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster2-client.mirror.129766.log: 89.8% -- replaced with /var/log/ceph/cluster2-client.mirror.129766.log.gz 2023-08-26T18:17:48.991 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.125627.log 2023-08-26T18:17:48.991 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster1-client.mirror.125627.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.125627.log.gz 2023-08-26T18:17:48.991 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.114617.log 2023-08-26T18:17:48.991 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster2-client.mirror.114617.log: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.114617.log.gz 2023-08-26T18:17:48.992 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.141291.log 2023-08-26T18:17:48.992 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster2-client.mirror.141291.log: 88.1% -- replaced with /var/log/ceph/cluster2-client.mirror.141291.log.gz 2023-08-26T18:17:48.992 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.129607.log 2023-08-26T18:17:48.992 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster2-client.mirror.129607.log: 87.3% -- replaced with /var/log/ceph/cluster2-client.mirror.129607.log.gz 2023-08-26T18:17:48.993 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.123045.log 2023-08-26T18:17:48.994 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster1-client.mirror.123045.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.123045.log.gz 2023-08-26T18:17:48.995 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.116241.log 2023-08-26T18:17:48.995 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster2-client.mirror.116241.log: 79.6% -- replaced with /var/log/ceph/cluster2-client.mirror.116241.log.gz 2023-08-26T18:17:48.995 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.144797.log 2023-08-26T18:17:48.996 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.108412.log 2023-08-26T18:17:48.996 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster2-client.mirror.144797.log: 10.2% -- replaced with /var/log/ceph/cluster2-client.mirror.144797.log.gz 2023-08-26T18:17:48.996 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.132106.log 2023-08-26T18:17:48.997 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster2-client.mirror.108412.log: /var/log/ceph/cluster2-client.mirror.132106.log: 89.9% -- replaced with /var/log/ceph/cluster2-client.mirror.108412.log.gz 2023-08-26T18:17:48.997 INFO:teuthology.orchestra.run.smithi124.stderr: 89.7% -- replaced with /var/log/ceph/cluster2-client.mirror.132106.log.gz 2023-08-26T18:17:48.997 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.114179.log 2023-08-26T18:17:48.998 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster1-client.mirror.114179.log: 93.4% -- replaced with /var/log/ceph/cluster1-client.mirror.114179.log.gz 2023-08-26T18:17:48.998 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.142128.log 2023-08-26T18:17:48.999 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster2-client.mirror.142128.log: 87.4% -- replaced with /var/log/ceph/cluster2-client.mirror.142128.log.gz 2023-08-26T18:17:48.999 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.105422.log 2023-08-26T18:17:49.000 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.141920.log 2023-08-26T18:17:49.000 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster2-client.admin.105422.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.105422.log.gz 2023-08-26T18:17:49.001 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.127296.log 2023-08-26T18:17:49.001 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster1-client.mirror.141920.log: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.141920.log.gz 2023-08-26T18:17:49.002 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster2-client.mirror.127296.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.109942.log 2023-08-26T18:17:49.002 INFO:teuthology.orchestra.run.smithi124.stderr: 89.9% -- replaced with /var/log/ceph/cluster2-client.mirror.127296.log.gz 2023-08-26T18:17:49.003 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster1-client.mirror.109942.log: 87.6% -- replaced with /var/log/ceph/cluster1-client.mirror.109942.log.gz 2023-08-26T18:17:49.003 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.130186.log 2023-08-26T18:17:49.004 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster2-client.mirror.130186.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.132173.log 2023-08-26T18:17:49.004 INFO:teuthology.orchestra.run.smithi124.stderr: 87.4% -- replaced with /var/log/ceph/cluster2-client.mirror.130186.log.gz 2023-08-26T18:17:49.008 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster2-client.mirror.132173.log: 89.8% -- replaced with /var/log/ceph/cluster2-client.mirror.132173.log.gz 2023-08-26T18:17:49.008 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.143646.log 2023-08-26T18:17:49.009 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster2-client.admin.143646.log: 8.2% -- replaced with /var/log/ceph/cluster2-client.admin.143646.log.gz 2023-08-26T18:17:49.009 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.121008.log 2023-08-26T18:17:49.010 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.125945.log 2023-08-26T18:17:49.010 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster1-client.mirror.121008.log: /var/log/ceph/cluster1-client.mirror.125945.log: 87.6% -- replaced with /var/log/ceph/cluster1-client.mirror.125945.log.gz 2023-08-26T18:17:49.011 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.134967.log 2023-08-26T18:17:49.011 INFO:teuthology.orchestra.run.smithi124.stderr: 92.2% -- replaced with /var/log/ceph/cluster1-client.mirror.121008.log.gz 2023-08-26T18:17:49.012 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster2-client.mirror.134967.log: 88.0% -- replaced with /var/log/ceph/cluster2-client.mirror.134967.log.gz 2023-08-26T18:17:49.012 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.117782.log 2023-08-26T18:17:49.013 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster2-client.mirror.117782.log: 79.4% -- replaced with /var/log/ceph/cluster2-client.mirror.117782.log.gz 2023-08-26T18:17:49.013 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.134325.log 2023-08-26T18:17:49.017 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster2-client.mirror.134325.log: 90.4% -- replaced with /var/log/ceph/cluster2-client.mirror.134325.log.gz 2023-08-26T18:17:49.017 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.110852.log 2023-08-26T18:17:49.017 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster1-client.mirror.110852.log: 88.4% -- replaced with /var/log/ceph/cluster1-client.mirror.110852.log.gz 2023-08-26T18:17:49.018 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.138658.log 2023-08-26T18:17:49.019 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.138320.log 2023-08-26T18:17:49.019 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster1-client.mirror.138658.log: 86.8% -- replaced with /var/log/ceph/cluster1-client.mirror.138658.log.gz 2023-08-26T18:17:49.020 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster1-client.mirror.138320.log: 87.0% -- replaced with /var/log/ceph/cluster1-client.mirror.138320.log.gz 2023-08-26T18:17:49.020 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.112741.log 2023-08-26T18:17:49.022 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster1-client.admin.112741.log: 88.3% -- replaced with /var/log/ceph/cluster1-client.admin.112741.log.gz 2023-08-26T18:17:49.022 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.122710.log 2023-08-26T18:17:49.022 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster2-client.mirror.122710.log: 89.7% -- replaced with /var/log/ceph/cluster2-client.mirror.122710.log.gz 2023-08-26T18:17:49.022 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.131763.log 2023-08-26T18:17:49.023 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster1-client.mirror.131763.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.131763.log.gz 2023-08-26T18:17:49.023 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.123158.log 2023-08-26T18:17:49.025 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster1-client.mirror.123158.log: 92.2% -- replaced with /var/log/ceph/cluster1-client.mirror.123158.log.gz 2023-08-26T18:17:49.025 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.121175.log 2023-08-26T18:17:49.026 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster1-client.mirror.121175.log: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.121175.log.gz 2023-08-26T18:17:49.026 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.114889.log 2023-08-26T18:17:49.027 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster2-client.mirror.114889.log: 79.5% -- replaced with /var/log/ceph/cluster2-client.mirror.114889.log.gz 2023-08-26T18:17:49.027 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.136147.log 2023-08-26T18:17:49.028 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster2-client.mirror.136147.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.137915.log 2023-08-26T18:17:49.030 INFO:teuthology.orchestra.run.smithi124.stderr: 90.4% -- replaced with /var/log/ceph/cluster2-client.mirror.136147.log.gz 2023-08-26T18:17:49.033 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster1-client.mirror.137915.log: 86.6% -- replaced with /var/log/ceph/cluster1-client.mirror.137915.log.gz 2023-08-26T18:17:49.033 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.109348.log 2023-08-26T18:17:49.035 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster2-client.mirror.109348.log: 89.8% -- replaced with /var/log/ceph/cluster2-client.mirror.109348.log.gz 2023-08-26T18:17:49.035 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.136052.log 2023-08-26T18:17:49.036 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster2-client.mirror.136052.log: 88.0% -- replaced with /var/log/ceph/cluster2-client.mirror.136052.log.gz 2023-08-26T18:17:49.037 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.130884.log 2023-08-26T18:17:49.038 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster2-client.mirror.130884.log: 89.8% -- replaced with /var/log/ceph/cluster2-client.mirror.130884.log.gz 2023-08-26T18:17:49.038 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.142598.log 2023-08-26T18:17:49.039 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster1-client.mirror.142598.log: 86.8% -- replaced with /var/log/ceph/cluster1-client.mirror.142598.log.gz 2023-08-26T18:17:49.039 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.118172.log 2023-08-26T18:17:49.040 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster2-client.mirror.118172.log: 89.7% -- replaced with /var/log/ceph/cluster2-client.mirror.118172.log.gz 2023-08-26T18:17:49.040 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.145264.log 2023-08-26T18:17:49.042 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster2-client.mirror.145264.log: 9.2% -- replaced with /var/log/ceph/cluster2-client.mirror.145264.log.gz 2023-08-26T18:17:49.042 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.123069.log 2023-08-26T18:17:49.042 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster1-client.mirror.123069.log: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.123069.log.gz 2023-08-26T18:17:49.043 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.117574.log 2023-08-26T18:17:49.044 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster2-client.mirror.117574.log: 87.4% -- replaced with /var/log/ceph/cluster2-client.mirror.117574.log.gz 2023-08-26T18:17:49.044 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.142985.log 2023-08-26T18:17:49.045 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster1-client.mirror.142985.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.142985.log.gz 2023-08-26T18:17:49.045 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.118068.log 2023-08-26T18:17:49.045 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster2-client.mirror.118068.log: 79.5% -- replaced with /var/log/ceph/cluster2-client.mirror.118068.log.gz 2023-08-26T18:17:49.045 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.123599.log 2023-08-26T18:17:49.046 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster1-client.mirror.123599.log: 87.6% -- replaced with /var/log/ceph/cluster1-client.mirror.123599.log.gz 2023-08-26T18:17:49.047 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.117720.log 2023-08-26T18:17:49.048 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.0.131574.log 2023-08-26T18:17:49.050 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster2-client.mirror.117720.log: /var/log/ceph/cluster2-client.mirror.0.131574.log: 89.6% -- replaced with /var/log/ceph/cluster2-client.mirror.117720.log.gz 2023-08-26T18:17:49.054 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.143006.log 2023-08-26T18:17:49.054 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster1-client.mirror.143006.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.143006.log.gz 2023-08-26T18:17:49.065 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.118279.log 2023-08-26T18:17:49.066 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster2-client.mirror.118279.log: 87.5% -- replaced with /var/log/ceph/cluster2-client.mirror.118279.log.gz 2023-08-26T18:17:49.071 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.1.141004.log 2023-08-26T18:17:49.082 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster2-client.mirror.1.141004.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.136747.log 2023-08-26T18:17:49.083 INFO:teuthology.orchestra.run.smithi124.stderr: 93.8% -- replaced with /var/log/ceph/cluster2-client.mirror.110398.log.gz 2023-08-26T18:17:49.087 INFO:teuthology.orchestra.run.smithi124.stderr: 93.0% -- replaced with /var/log/ceph/cluster2-client.mirror.0.131574.log.gz 2023-08-26T18:17:49.088 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster2-client.mirror.136747.log: 88.1% -- replaced with /var/log/ceph/cluster2-client.mirror.136747.log.gz 2023-08-26T18:17:49.088 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.129423.log 2023-08-26T18:17:49.090 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster1-client.mirror.129423.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.129423.log.gz 2023-08-26T18:17:49.090 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.145481.log 2023-08-26T18:17:49.091 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster2-client.mirror.145481.log: 6.1% -- replaced with /var/log/ceph/cluster2-client.mirror.145481.log.gz 2023-08-26T18:17:49.091 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.105054.log 2023-08-26T18:17:49.093 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster2-client.admin.105054.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.105054.log.gz 2023-08-26T18:17:49.094 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.142619.log 2023-08-26T18:17:49.094 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster1-client.admin.142619.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.142619.log.gz 2023-08-26T18:17:49.095 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.117416.log 2023-08-26T18:17:49.097 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster2-client.mirror.117416.log: 93.7% -- replaced with /var/log/ceph/cluster2-client.mirror.118842.log.gz 2023-08-26T18:17:49.097 INFO:teuthology.orchestra.run.smithi124.stderr: 87.5% -- replaced with /var/log/ceph/cluster2-client.mirror.117416.log.gz 2023-08-26T18:17:49.097 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.110291.log 2023-08-26T18:17:49.098 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.140473.log 2023-08-26T18:17:49.098 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster2-client.mirror.110291.log: /var/log/ceph/cluster2-client.mirror.140473.log: 79.6% -- replaced with /var/log/ceph/cluster2-client.mirror.140473.log.gz 2023-08-26T18:17:49.098 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.144268.log 2023-08-26T18:17:49.099 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster2-client.mirror.144268.log: 8.2% -- replaced with /var/log/ceph/cluster2-client.mirror.144268.log.gz 2023-08-26T18:17:49.099 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.111600.log 2023-08-26T18:17:49.101 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster1-client.mirror.111600.log: 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.111600.log.gz 2023-08-26T18:17:49.101 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.142805.log 2023-08-26T18:17:49.102 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster1-client.mirror.142805.log: 67.9% -- replaced with /var/log/ceph/cluster1-client.mirror.142805.log.gz 2023-08-26T18:17:49.102 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.136031.log 2023-08-26T18:17:49.102 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster2-client.mirror.136031.log: 93.4% -- replaced with /var/log/ceph/cluster2-client.mirror.110291.log.gz 2023-08-26T18:17:49.102 INFO:teuthology.orchestra.run.smithi124.stderr: 87.9% -- replaced with /var/log/ceph/cluster2-client.mirror.136031.log.gz 2023-08-26T18:17:49.103 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.0.124810.log 2023-08-26T18:17:49.103 INFO:teuthology.orchestra.run.smithi124.stderr: 95.5% -- replaced with /var/log/ceph/cluster2-client.mirror.1.141004.log.gz 2023-08-26T18:17:49.103 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.124992.log 2023-08-26T18:17:49.104 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.120240.log 2023-08-26T18:17:49.106 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster2-client.mirror.0.124810.log: /var/log/ceph/cluster2-client.mirror.124992.log: /var/log/ceph/cluster1-client.mirror.120240.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.120240.log.gz 2023-08-26T18:17:49.106 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.117495.log 2023-08-26T18:17:49.106 INFO:teuthology.orchestra.run.smithi124.stderr: 92.4% -- replaced with /var/log/ceph/cluster2-client.mirror.124992.log.gz 2023-08-26T18:17:49.107 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.113835.log 2023-08-26T18:17:49.107 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster2-client.mirror.117495.log: /var/log/ceph/cluster1-client.mirror.113835.log: 87.5% -- replaced with /var/log/ceph/cluster2-client.mirror.117495.log.gz 2023-08-26T18:17:49.107 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.131533.log 2023-08-26T18:17:49.107 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.118696.log 2023-08-26T18:17:49.108 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster2-client.mirror.131533.log: /var/log/ceph/cluster2-client.mirror.118696.log: 79.6% -- replaced with /var/log/ceph/cluster2-client.mirror.118696.log.gz 2023-08-26T18:17:49.108 INFO:teuthology.orchestra.run.smithi124.stderr: 89.7% -- replaced with /var/log/ceph/cluster2-client.mirror.131533.log.gz 2023-08-26T18:17:49.108 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.109186.log 2023-08-26T18:17:49.110 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster1-client.mirror.109186.log: 87.8% -- replaced with /var/log/ceph/cluster1-client.mirror.109186.log.gz 2023-08-26T18:17:49.110 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.118105.log 2023-08-26T18:17:49.110 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster2-client.mirror.118105.log: 87.4% -- replaced with /var/log/ceph/cluster2-client.mirror.118105.log.gz 2023-08-26T18:17:49.111 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.128965.log 2023-08-26T18:17:49.111 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster1-client.mirror.128965.log: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.128965.log.gz 2023-08-26T18:17:49.111 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.130303.log 2023-08-26T18:17:49.111 INFO:teuthology.orchestra.run.smithi124.stderr: 93.2% -- replaced with /var/log/ceph/cluster1-client.mirror.113835.log.gz 2023-08-26T18:17:49.113 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster2-client.mirror.130303.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.145783.log 2023-08-26T18:17:49.113 INFO:teuthology.orchestra.run.smithi124.stderr: 92.4% -- replaced with /var/log/ceph/cluster2-client.mirror.130303.log.gz 2023-08-26T18:17:49.113 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.108182.log 2023-08-26T18:17:49.114 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster2-client.admin.145783.log: 94.0% -- replaced with /var/log/ceph/cluster2-client.mirror.0.124810.log.gz 2023-08-26T18:17:49.114 INFO:teuthology.orchestra.run.smithi124.stderr: 8.2% -- replaced with /var/log/ceph/cluster2-client.admin.145783.log.gz 2023-08-26T18:17:49.114 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster1-client.mirror.108182.log: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.108182.log.gz 2023-08-26T18:17:49.115 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.113749.log 2023-08-26T18:17:49.116 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.131000.log 2023-08-26T18:17:49.117 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster1-client.mirror.113749.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.140490.log 2023-08-26T18:17:49.117 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster1-client.mirror.131000.log: 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.131000.log.gz 2023-08-26T18:17:49.118 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.138050.log 2023-08-26T18:17:49.118 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster2-client.mirror.140490.log: /var/log/ceph/cluster1-client.mirror.138050.log: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.138050.log.gz 2023-08-26T18:17:49.119 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.111626.log 2023-08-26T18:17:49.119 INFO:teuthology.orchestra.run.smithi124.stderr: 89.6% -- replaced with /var/log/ceph/cluster2-client.mirror.140490.log.gz 2023-08-26T18:17:49.129 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.99963.log 2023-08-26T18:17:49.130 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster2-client.mirror.111626.log: /var/log/ceph/cluster2-client.admin.99963.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.99963.log.gz 2023-08-26T18:17:49.130 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2.log 2023-08-26T18:17:49.131 INFO:teuthology.orchestra.run.smithi124.stderr: 91.7% -- replaced with /var/log/ceph/cluster2-client.mirror.111626.log.gz 2023-08-26T18:17:49.131 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.117616.log 2023-08-26T18:17:49.131 INFO:teuthology.orchestra.run.smithi124.stderr: 93.4% -- replaced with /var/log/ceph/cluster1-client.mirror.113749.log.gz 2023-08-26T18:17:49.132 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster2.log: /var/log/ceph/cluster2-client.mirror.117616.log: 79.8% -- replaced with /var/log/ceph/cluster2-client.mirror.117616.log.gz 2023-08-26T18:17:49.132 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.132943.log 2023-08-26T18:17:49.132 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster1-client.mirror.132943.log: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.132943.log.gz 2023-08-26T18:17:49.133 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.143301.log 2023-08-26T18:17:49.133 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.125246.log 2023-08-26T18:17:49.134 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster1-client.mirror.143301.log: 1.2% -- replaced with /var/log/ceph/cluster1-client.mirror.143301.log.gz 2023-08-26T18:17:49.134 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster1-client.mirror.125246.log: 92.3% -- replaced with /var/log/ceph/cluster1-client.mirror.125246.log.gz 2023-08-26T18:17:49.134 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.141218.log 2023-08-26T18:17:49.134 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.108371.log 2023-08-26T18:17:49.135 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster2-client.mirror.141218.log: 87.6% -- replaced with /var/log/ceph/cluster2.log.gz 2023-08-26T18:17:49.135 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster2-client.mirror.108371.log: 88.6% -- replaced with /var/log/ceph/cluster2-client.mirror.141218.log.gz 2023-08-26T18:17:49.135 INFO:teuthology.orchestra.run.smithi124.stderr: 91.6% -- replaced with /var/log/ceph/cluster2-client.mirror.108371.log.gz 2023-08-26T18:17:49.136 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.137533.log 2023-08-26T18:17:49.136 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.129724.log 2023-08-26T18:17:49.136 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster1-client.mirror.137533.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.105560.log 2023-08-26T18:17:49.136 INFO:teuthology.orchestra.run.smithi124.stderr: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.137533.log.gz 2023-08-26T18:17:49.137 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster2-client.mirror.129724.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.122551.log 2023-08-26T18:17:49.140 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster1-client.admin.105560.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.105560.log.gz 2023-08-26T18:17:49.140 INFO:teuthology.orchestra.run.smithi124.stderr: 92.5% -- replaced with /var/log/ceph/cluster2-client.mirror.129724.log.gz 2023-08-26T18:17:49.140 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.129631.log 2023-08-26T18:17:49.141 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster2-client.mirror.122551.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.132019.log 2023-08-26T18:17:49.141 INFO:teuthology.orchestra.run.smithi124.stderr: 87.4% -- replaced with /var/log/ceph/cluster2-client.mirror.122551.log.gz 2023-08-26T18:17:49.141 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster1-client.mirror.129631.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.142070.log 2023-08-26T18:17:49.142 INFO:teuthology.orchestra.run.smithi124.stderr: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.129631.log.gz 2023-08-26T18:17:49.142 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster1-client.mirror.132019.log: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.132019.log.gz 2023-08-26T18:17:49.142 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.116653.log 2023-08-26T18:17:49.142 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster2-client.mirror.142070.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.139122.log 2023-08-26T18:17:49.143 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster2-client.mirror.116653.log: 91.6% -- replaced with /var/log/ceph/cluster2-client.mirror.142070.log.gz 2023-08-26T18:17:49.143 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.119452.log 2023-08-26T18:17:49.143 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster2-client.mirror.139122.log: 79.4% -- replaced with /var/log/ceph/cluster2-client.mirror.139122.log.gz 2023-08-26T18:17:49.144 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster2-client.mirror.119452.log: 89.7% -- replaced with /var/log/ceph/cluster2-client.mirror.119452.log.gz 2023-08-26T18:17:49.144 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.116552.log 2023-08-26T18:17:49.144 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.144219.log 2023-08-26T18:17:49.144 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster2-client.mirror.116552.log: /var/log/ceph/cluster2-client.mirror.144219.log: 87.4% -- replaced with /var/log/ceph/cluster2-client.mirror.116552.log.gz 2023-08-26T18:17:49.145 INFO:teuthology.orchestra.run.smithi124.stderr: 6.1% -- replaced with /var/log/ceph/cluster2-client.mirror.144219.log.gz 2023-08-26T18:17:49.145 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.115548.log 2023-08-26T18:17:49.145 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.3.124621.log 2023-08-26T18:17:49.145 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster2-client.mirror.115548.log: /var/log/ceph/cluster1-client.mirror.3.124621.log: 87.4% -- replaced with /var/log/ceph/cluster2-client.mirror.115548.log.gz 2023-08-26T18:17:49.146 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.143339.log 2023-08-26T18:17:49.146 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster1-client.mirror.143339.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.143339.log.gz 2023-08-26T18:17:49.146 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.133460.log 2023-08-26T18:17:49.149 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster2-client.mirror.133460.log: 89.8% -- replaced with /var/log/ceph/cluster2-client.mirror.133460.log.gz 2023-08-26T18:17:49.151 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.109490.log 2023-08-26T18:17:49.152 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster2-client.mirror.109490.log: 73.5% -- replaced with /var/log/ceph/cluster2-client.mirror.109490.log.gz 2023-08-26T18:17:49.152 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.143593.log 2023-08-26T18:17:49.152 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster2-client.mirror.143593.log: 8.2% -- replaced with /var/log/ceph/cluster2-client.mirror.143593.log.gz 2023-08-26T18:17:49.152 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.113025.log 2023-08-26T18:17:49.153 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster2-client.mirror.113025.log: 90.1% -- replaced with /var/log/ceph/cluster2-client.mirror.113025.log.gz 2023-08-26T18:17:49.153 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.109450.log 2023-08-26T18:17:49.153 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster2-client.mirror.109450.log: 91.4% -- replaced with /var/log/ceph/cluster2-client.mirror.109450.log.gz 2023-08-26T18:17:49.154 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.142008.log 2023-08-26T18:17:49.154 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster1-client.mirror.142008.log: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.142008.log.gz 2023-08-26T18:17:49.154 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.131041.log 2023-08-26T18:17:49.154 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster1-client.mirror.131041.log: 92.0% -- replaced with /var/log/ceph/cluster1-client.mirror.131041.log.gz 2023-08-26T18:17:49.155 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.128223.log 2023-08-26T18:17:49.155 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster1-client.mirror.128223.log: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.128223.log.gz 2023-08-26T18:17:49.155 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.125946.log 2023-08-26T18:17:49.156 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster2-client.mirror.125946.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.119783.log 2023-08-26T18:17:49.157 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster2-client.mirror.119783.log: 79.4% -- replaced with /var/log/ceph/cluster2-client.mirror.119783.log.gz 2023-08-26T18:17:49.157 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.112419.log 2023-08-26T18:17:49.158 INFO:teuthology.orchestra.run.smithi124.stderr: 93.7% -- replaced with /var/log/ceph/cluster2-client.mirror.119247.log.gz 2023-08-26T18:17:49.158 INFO:teuthology.orchestra.run.smithi124.stderr: 93.5% -- replaced with /var/log/ceph/cluster2-client.mirror.125946.log.gz 2023-08-26T18:17:49.158 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.128727.log 2023-08-26T18:17:49.158 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster1-client.mirror.112419.log: /var/log/ceph/cluster1-client.mirror.128727.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.128727.log.gz 2023-08-26T18:17:49.159 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.0.110441.log 2023-08-26T18:17:49.163 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster1-client.mirror.0.110441.log: 93.3% -- replaced with /var/log/ceph/cluster1-client.mirror.112419.log.gz 2023-08-26T18:17:49.163 INFO:teuthology.orchestra.run.smithi124.stderr: 93.2% -- replaced with /var/log/ceph/cluster1-client.mirror.0.110441.log.gz 2023-08-26T18:17:49.164 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.125484.log 2023-08-26T18:17:49.164 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.108925.log 2023-08-26T18:17:49.164 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster1-client.mirror.125484.log: /var/log/ceph/cluster1-client.admin.108925.log: 87.5% -- replaced with /var/log/ceph/cluster1-client.admin.108925.log.gz 2023-08-26T18:17:49.165 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.135132.log 2023-08-26T18:17:49.165 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/cluster1-mgr.x.log: 91.4% -- replaced with /var/log/ceph/cluster2-mgr.x.log.gz 2023-08-26T18:17:49.167 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster2-client.mirror.135132.log: 92.4% -- replaced with /var/log/ceph/cluster1-client.mirror.125484.log.gz 2023-08-26T18:17:49.167 INFO:teuthology.orchestra.run.smithi124.stderr: 90.4% -- replaced with /var/log/ceph/cluster2-client.mirror.135132.log.gz 2023-08-26T18:17:49.167 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.110354.log 2023-08-26T18:17:49.168 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.115297.log 2023-08-26T18:17:49.169 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster1-client.mirror.110354.log: /var/log/ceph/cluster2-client.mirror.115297.log: 79.4% -- replaced with /var/log/ceph/cluster2-client.mirror.115297.log.gz 2023-08-26T18:17:49.169 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.113643.log 2023-08-26T18:17:49.171 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster2-client.mirror.113643.log: 93.4% -- replaced with /var/log/ceph/cluster1-client.mirror.110354.log.gz 2023-08-26T18:17:49.173 INFO:teuthology.orchestra.run.smithi124.stderr: 93.4% -- replaced with /var/log/ceph/cluster2-client.mirror.113643.log.gz 2023-08-26T18:17:49.173 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.129278.log 2023-08-26T18:17:49.174 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.124211.log 2023-08-26T18:17:49.175 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster2-client.mirror.129278.log: /var/log/ceph/cluster2-client.mirror.124211.log: 87.3% -- replaced with /var/log/ceph/cluster2-client.mirror.124211.log.gz 2023-08-26T18:17:49.175 INFO:teuthology.orchestra.run.smithi124.stderr: 89.7% -- replaced with /var/log/ceph/cluster2-client.mirror.129278.log.gz 2023-08-26T18:17:49.176 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.105307.log 2023-08-26T18:17:49.176 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.120615.log 2023-08-26T18:17:49.176 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster2-client.admin.105307.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.105307.log.gz 2023-08-26T18:17:49.177 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster2-client.mirror.120615.log: 87.3% -- replaced with /var/log/ceph/cluster2-client.mirror.120615.log.gz 2023-08-26T18:17:49.177 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.120177.log 2023-08-26T18:17:49.178 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.136705.log 2023-08-26T18:17:49.178 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster2-client.mirror.120177.log: 1.3% -- replaced with /var/log/ceph/cluster2-client.mirror.120177.log.gz 2023-08-26T18:17:49.179 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster2-client.mirror.136705.log: 90.4% -- replaced with /var/log/ceph/cluster2-client.mirror.136705.log.gz 2023-08-26T18:17:49.179 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.135651.log 2023-08-26T18:17:49.180 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.142271.log 2023-08-26T18:17:49.181 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster2-client.mirror.135651.log: /var/log/ceph/cluster1-client.mirror.142271.log: 88.0% -- replaced with /var/log/ceph/cluster2-client.mirror.135651.log.gz 2023-08-26T18:17:49.181 INFO:teuthology.orchestra.run.smithi124.stderr: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.142271.log.gz 2023-08-26T18:17:49.181 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.116769.log 2023-08-26T18:17:49.181 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.114948.log 2023-08-26T18:17:49.182 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster2-client.mirror.116769.log: /var/log/ceph/cluster2-client.mirror.114948.log: 89.6% -- replaced with /var/log/ceph/cluster2-client.mirror.116769.log.gz 2023-08-26T18:17:49.187 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.111666.log 2023-08-26T18:17:49.188 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster1-client.mirror.111666.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.125419.log 2023-08-26T18:17:49.189 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster1-client.mirror.125419.log: 87.2% -- replaced with /var/log/ceph/cluster1-client.mirror.125419.log.gz 2023-08-26T18:17:49.190 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.123896.log 2023-08-26T18:17:49.191 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster1-client.mirror.123896.log: 92.3% -- replaced with /var/log/ceph/cluster1-client.mirror.123896.log.gz 2023-08-26T18:17:49.191 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.115209.log 2023-08-26T18:17:49.192 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster2-client.mirror.115209.log: 79.4% -- replaced with /var/log/ceph/cluster2-client.mirror.115209.log.gz 2023-08-26T18:17:49.192 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.144849.log 2023-08-26T18:17:49.192 INFO:teuthology.orchestra.run.smithi124.stderr: 95.1% -- replaced with /var/log/ceph/cluster1-client.mirror.111666.log.gz 2023-08-26T18:17:49.193 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster2-client.mirror.144849.log: 10.2% -- replaced with /var/log/ceph/cluster2-client.mirror.144849.log.gz 2023-08-26T18:17:49.194 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.104915.log 2023-08-26T18:17:49.194 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster2-client.admin.104915.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.104915.log.gz 2023-08-26T18:17:49.194 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.104686.log 2023-08-26T18:17:49.195 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster2-client.admin.104686.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.104686.log.gz 2023-08-26T18:17:49.196 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.99738.log 2023-08-26T18:17:49.196 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster2-client.admin.99738.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.99738.log.gz 2023-08-26T18:17:49.196 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.117633.log 2023-08-26T18:17:49.197 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster2-client.mirror.117633.log: 89.7% -- replaced with /var/log/ceph/cluster2-client.mirror.117633.log.gz 2023-08-26T18:17:49.197 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.142248.log 2023-08-26T18:17:49.198 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster1-client.mirror.142248.log: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.142248.log.gz 2023-08-26T18:17:49.198 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.109162.log 2023-08-26T18:17:49.199 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster1-client.mirror.109162.log: 87.9% -- replaced with /var/log/ceph/cluster1-client.mirror.109162.log.gz 2023-08-26T18:17:49.199 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.127953.log 2023-08-26T18:17:49.200 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster1-client.mirror.127953.log: 92.4% -- replaced with /var/log/ceph/cluster1-client.mirror.127953.log.gz 2023-08-26T18:17:49.200 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.133900.log 2023-08-26T18:17:49.202 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster2-client.mirror.133900.log: 87.9% -- replaced with /var/log/ceph/cluster2-client.mirror.133900.log.gz 2023-08-26T18:17:49.202 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.111052.log 2023-08-26T18:17:49.203 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster1-client.admin.111052.log: 88.0% -- replaced with /var/log/ceph/cluster1-client.admin.111052.log.gz 2023-08-26T18:17:49.203 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.132723.log 2023-08-26T18:17:49.204 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster1-client.mirror.132723.log: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.132723.log.gz 2023-08-26T18:17:49.204 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.122279.log 2023-08-26T18:17:49.205 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster2-client.mirror.122279.log: 89.8% -- replaced with /var/log/ceph/cluster2-client.mirror.122279.log.gz 2023-08-26T18:17:49.205 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.136209.log 2023-08-26T18:17:49.207 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.130210.log 2023-08-26T18:17:49.208 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster2-client.mirror.136209.log: /var/log/ceph/cluster1-client.mirror.130210.log: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.130210.log.gz 2023-08-26T18:17:49.208 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.128042.log 2023-08-26T18:17:49.209 INFO:teuthology.orchestra.run.smithi124.stderr: 90.4% -- replaced with /var/log/ceph/cluster2-client.mirror.136209.log.gz 2023-08-26T18:17:49.209 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster2-client.mirror.128042.log: 89.9% -- replaced with /var/log/ceph/cluster2-client.mirror.128042.log.gz 2023-08-26T18:17:49.210 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.136168.log 2023-08-26T18:17:49.210 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster2-client.mirror.136168.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.141857.log 2023-08-26T18:17:49.211 INFO:teuthology.orchestra.run.smithi124.stderr: 88.0% -- replaced with /var/log/ceph/cluster2-client.mirror.136168.log.gz 2023-08-26T18:17:49.216 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster2-client.mirror.141857.log: 89.6% -- replaced with /var/log/ceph/cluster2-client.mirror.141857.log.gz 2023-08-26T18:17:49.216 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.130617.log 2023-08-26T18:17:49.218 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster1-client.mirror.130617.log: 87.6% -- replaced with /var/log/ceph/cluster1-client.mirror.130617.log.gz 2023-08-26T18:17:49.219 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.124051.log 2023-08-26T18:17:49.219 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster1-client.mirror.124051.log: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.124051.log.gz 2023-08-26T18:17:49.220 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.143138.log 2023-08-26T18:17:49.221 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster1-client.mirror.143138.log: 2.3% -- replaced with /var/log/ceph/cluster1-client.mirror.143138.log.gz 2023-08-26T18:17:49.221 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.2.117290.log 2023-08-26T18:17:49.226 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster2-client.mirror.2.117290.log: 93.8% -- replaced with /var/log/ceph/cluster1-client.mirror.3.124621.log.gz 2023-08-26T18:17:49.227 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.135112.log 2023-08-26T18:17:49.228 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster2-client.mirror.135112.log: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.135112.log.gz 2023-08-26T18:17:49.228 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.115621.log 2023-08-26T18:17:49.230 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster2-client.mirror.115621.log: 89.7% -- replaced with /var/log/ceph/cluster2-client.mirror.115621.log.gz 2023-08-26T18:17:49.230 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.119820.log 2023-08-26T18:17:49.231 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster2-client.mirror.119820.log: 87.4% -- replaced with /var/log/ceph/cluster2-client.mirror.119820.log.gz 2023-08-26T18:17:49.232 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.106901.log 2023-08-26T18:17:49.234 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster2-client.mirror.106901.log: 57.4% -- replaced with /var/log/ceph/cluster2-client.mirror.106901.log.gz 2023-08-26T18:17:49.235 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.130593.log 2023-08-26T18:17:49.235 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster1-client.mirror.130593.log: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.130593.log.gz 2023-08-26T18:17:49.235 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.113792.log 2023-08-26T18:17:49.237 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster1-client.mirror.113792.log: 93.9% -- replaced with /var/log/ceph/cluster2-client.mirror.2.117290.log.gz 2023-08-26T18:17:49.238 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.111459.log 2023-08-26T18:17:49.240 INFO:teuthology.orchestra.run.smithi031.stderr: 91.1% -- replaced with /var/log/ceph/cluster1-mgr.x.log.gz 2023-08-26T18:17:49.240 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster1-client.mirror.111459.log: 93.3% -- replaced with /var/log/ceph/cluster1-client.mirror.113792.log.gz 2023-08-26T18:17:49.241 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.116163.log 2023-08-26T18:17:49.241 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster2-client.mirror.116163.log: 89.7% -- replaced with /var/log/ceph/cluster2-client.mirror.116163.log.gz 2023-08-26T18:17:49.242 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.114517.log 2023-08-26T18:17:49.243 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster2-client.mirror.114517.log: 88.4% -- replaced with /var/log/ceph/cluster2-client.mirror.114517.log.gz 2023-08-26T18:17:49.243 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.116673.log 2023-08-26T18:17:49.244 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster2-client.mirror.116673.log: 93.4% -- replaced with /var/log/ceph/cluster1-client.mirror.111459.log.gz 2023-08-26T18:17:49.244 INFO:teuthology.orchestra.run.smithi124.stderr: 79.5% -- replaced with /var/log/ceph/cluster2-client.mirror.116673.log.gz 2023-08-26T18:17:49.244 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.111028.log 2023-08-26T18:17:49.254 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.117979.log 2023-08-26T18:17:49.254 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster2-client.mirror.111028.log: /var/log/ceph/cluster2-client.mirror.117979.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.117554.log 2023-08-26T18:17:49.254 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster2-client.mirror.117554.log: 89.6% -- replaced with /var/log/ceph/cluster2-client.mirror.117554.log.gz 2023-08-26T18:17:49.255 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.115528.log 2023-08-26T18:17:49.281 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster2-client.mirror.115528.log: 89.6% -- replaced with /var/log/ceph/cluster2-client.mirror.115528.log.gz 2023-08-26T18:17:49.281 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.126690.log 2023-08-26T18:17:49.282 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster1-client.mirror.126690.log: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.126690.log.gz 2023-08-26T18:17:49.282 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.124356.log 2023-08-26T18:17:49.283 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster1-client.mirror.124356.log: 92.2% -- replaced with /var/log/ceph/cluster1-client.mirror.124356.log.gz 2023-08-26T18:17:49.283 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.110376.log 2023-08-26T18:17:49.283 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster2-client.mirror.110376.log: 89.7% -- replaced with /var/log/ceph/cluster2-client.mirror.110376.log.gz 2023-08-26T18:17:49.284 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.142458.log 2023-08-26T18:17:49.284 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster1-client.mirror.142458.log: 86.7% -- replaced with /var/log/ceph/cluster1-client.mirror.142458.log.gz 2023-08-26T18:17:49.284 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.119047.log 2023-08-26T18:17:49.288 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster2-client.mirror.119047.log: 89.6% -- replaced with /var/log/ceph/cluster2-client.mirror.119047.log.gz 2023-08-26T18:17:49.288 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.135215.log 2023-08-26T18:17:49.288 INFO:teuthology.orchestra.run.smithi124.stderr: 93.5% -- replaced with /var/log/ceph/cluster2-client.mirror.117979.log.gz 2023-08-26T18:17:49.289 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster2-client.mirror.135215.log: 88.1% -- replaced with /var/log/ceph/cluster2-client.mirror.135215.log.gz 2023-08-26T18:17:49.290 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.144579.log 2023-08-26T18:17:49.290 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster2-client.mirror.144579.log: 9.2% -- replaced with /var/log/ceph/cluster2-client.mirror.144579.log.gz 2023-08-26T18:17:49.290 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.138950.log 2023-08-26T18:17:49.292 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster1-client.mirror.138950.log: 86.8% -- replaced with /var/log/ceph/cluster1-client.mirror.138950.log.gz 2023-08-26T18:17:49.293 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.123394.log 2023-08-26T18:17:49.294 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster2-client.mirror.123394.log: 93.7% -- replaced with /var/log/ceph/cluster2-client.mirror.120147.log.gz 2023-08-26T18:17:49.294 INFO:teuthology.orchestra.run.smithi124.stderr: 92.5% -- replaced with /var/log/ceph/cluster2-client.mirror.123394.log.gz 2023-08-26T18:17:49.295 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.129807.log 2023-08-26T18:17:49.297 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.130392.log 2023-08-26T18:17:49.297 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster1-client.mirror.129807.log: /var/log/ceph/cluster2-client.mirror.130392.log: 94.5% -- replaced with /var/log/ceph/cluster1-client.mirror.129807.log.gz 2023-08-26T18:17:49.297 INFO:teuthology.orchestra.run.smithi124.stderr: 89.8% -- replaced with /var/log/ceph/cluster2-client.mirror.130392.log.gz 2023-08-26T18:17:49.297 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.119605.log 2023-08-26T18:17:49.298 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.106609.log 2023-08-26T18:17:49.299 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster2-client.mirror.119605.log: /var/log/ceph/cluster1-client.admin.106609.log: 66.4% -- replaced with /var/log/ceph/cluster1-client.admin.106609.log.gz 2023-08-26T18:17:49.299 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.133012.log 2023-08-26T18:17:49.300 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster2-client.mirror.133012.log: 92.5% -- replaced with /var/log/ceph/cluster2-client.mirror.133012.log.gz 2023-08-26T18:17:49.300 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.123021.log 2023-08-26T18:17:49.301 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster2-client.mirror.123021.log: 87.3% -- replaced with /var/log/ceph/cluster2-client.mirror.123021.log.gz 2023-08-26T18:17:49.301 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.129119.log 2023-08-26T18:17:49.302 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.122918.log 2023-08-26T18:17:49.302 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster2-client.mirror.129119.log: 87.3% -- replaced with /var/log/ceph/cluster2-client.mirror.129119.log.gz 2023-08-26T18:17:49.302 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster2-client.mirror.122918.log: 93.7% -- replaced with /var/log/ceph/cluster2-client.mirror.115965.log.gz 2023-08-26T18:17:49.303 INFO:teuthology.orchestra.run.smithi124.stderr: 92.4% -- replaced with /var/log/ceph/cluster2-client.mirror.122918.log.gz 2023-08-26T18:17:49.304 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.108432.log 2023-08-26T18:17:49.304 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.113686.log 2023-08-26T18:17:49.305 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster2-client.mirror.108432.log: /var/log/ceph/cluster2-client.mirror.113686.log: 87.5% -- replaced with /var/log/ceph/cluster2-client.mirror.108432.log.gz 2023-08-26T18:17:49.309 INFO:teuthology.orchestra.run.smithi124.stderr: 93.4% -- replaced with /var/log/ceph/cluster2-client.mirror.113686.log.gz 2023-08-26T18:17:49.310 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.107977.log 2023-08-26T18:17:49.311 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.117458.log 2023-08-26T18:17:49.311 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster2-client.mirror.107977.log: /var/log/ceph/cluster2-client.mirror.117458.log: 87.3% -- replaced with /var/log/ceph/cluster2-client.mirror.107977.log.gz 2023-08-26T18:17:49.312 INFO:teuthology.orchestra.run.smithi124.stderr: 79.4% -- replaced with /var/log/ceph/cluster2-client.mirror.117458.log.gz 2023-08-26T18:17:49.312 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.135775.log 2023-08-26T18:17:49.312 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.121708.log 2023-08-26T18:17:49.313 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster2-client.mirror.135775.log: /var/log/ceph/cluster2-client.mirror.121708.log: 87.9% -- replaced with /var/log/ceph/cluster2-client.mirror.135775.log.gz 2023-08-26T18:17:49.313 INFO:teuthology.orchestra.run.smithi124.stderr: 92.4% -- replaced with /var/log/ceph/cluster2-client.mirror.121708.log.gz 2023-08-26T18:17:49.314 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.124457.log 2023-08-26T18:17:49.315 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.111842.log 2023-08-26T18:17:49.315 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster2-client.mirror.124457.log: /var/log/ceph/cluster1-client.mirror.111842.log: 87.4% -- replaced with /var/log/ceph/cluster2-client.mirror.124457.log.gz 2023-08-26T18:17:49.315 INFO:teuthology.orchestra.run.smithi124.stderr: 88.8% -- replaced with /var/log/ceph/cluster1-client.mirror.111842.log.gz 2023-08-26T18:17:49.316 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.131491.log 2023-08-26T18:17:49.316 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.116515.log 2023-08-26T18:17:49.318 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster2-client.mirror.131491.log: /var/log/ceph/cluster2-client.mirror.116515.log: 79.4% -- replaced with /var/log/ceph/cluster2-client.mirror.116515.log.gz 2023-08-26T18:17:49.318 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.118879.log 2023-08-26T18:17:49.318 INFO:teuthology.orchestra.run.smithi124.stderr: 92.4% -- replaced with /var/log/ceph/cluster2-client.mirror.131491.log.gz 2023-08-26T18:17:49.318 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.1.106050.log 2023-08-26T18:17:49.319 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster2-client.mirror.118879.log: /var/log/ceph/cluster1-client.mirror.1.106050.log: 89.7% -- replaced with /var/log/ceph/cluster2-client.mirror.118879.log.gz 2023-08-26T18:17:49.320 INFO:teuthology.orchestra.run.smithi124.stderr: 91.4% -- replaced with /var/log/ceph/cluster1-client.mirror.1.106050.log.gz 2023-08-26T18:17:49.320 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.137960.log 2023-08-26T18:17:49.320 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.129215.log 2023-08-26T18:17:49.321 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster1-client.mirror.137960.log: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.137960.log.gz 2023-08-26T18:17:49.322 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster1-client.mirror.129215.log: 87.6% -- replaced with /var/log/ceph/cluster1-client.mirror.129215.log.gz 2023-08-26T18:17:49.322 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.125012.log 2023-08-26T18:17:49.322 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.106124.log 2023-08-26T18:17:49.323 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster1-client.mirror.125012.log: /var/log/ceph/cluster2-client.mirror.106124.log: 92.1% -- replaced with /var/log/ceph/cluster1-client.mirror.125012.log.gz 2023-08-26T18:17:49.332 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.106525.log 2023-08-26T18:17:49.333 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster1-client.admin.106525.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.106525.log.gz 2023-08-26T18:17:49.334 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.133169.log 2023-08-26T18:17:49.334 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster1-client.mirror.133169.log: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.133169.log.gz 2023-08-26T18:17:49.334 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.110246.log 2023-08-26T18:17:49.336 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster1-client.mirror.110246.log: 88.2% -- replaced with /var/log/ceph/cluster1-client.mirror.110246.log.gz 2023-08-26T18:17:49.336 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.108315.log 2023-08-26T18:17:49.337 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster1-client.admin.108315.log: 87.2% -- replaced with /var/log/ceph/cluster1-client.admin.108315.log.gz 2023-08-26T18:17:49.337 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.128814.log 2023-08-26T18:17:49.339 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster2-client.mirror.128814.log: 89.6% -- replaced with /var/log/ceph/cluster2-client.mirror.128814.log.gz 2023-08-26T18:17:49.340 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.121615.log 2023-08-26T18:17:49.340 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster1-client.mirror.121615.log: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.121615.log.gz 2023-08-26T18:17:49.340 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.140923.log 2023-08-26T18:17:49.340 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster2-client.mirror.140923.log: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.140923.log.gz 2023-08-26T18:17:49.341 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.129978.log 2023-08-26T18:17:49.341 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster1-client.mirror.129978.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.129978.log.gz 2023-08-26T18:17:49.342 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.104640.log 2023-08-26T18:17:49.342 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.113600.log 2023-08-26T18:17:49.346 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.117897.log 2023-08-26T18:17:49.346 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster2-client.mirror.113600.log: /var/log/ceph/cluster2-client.mirror.117897.log: 89.7% -- replaced with /var/log/ceph/cluster2-client.mirror.117897.log.gz 2023-08-26T18:17:49.346 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.141899.log 2023-08-26T18:17:49.347 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster2-client.mirror.141899.log: 93.4% -- replaced with /var/log/ceph/cluster2-client.mirror.113600.log.gz 2023-08-26T18:17:49.347 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster2-client.admin.104640.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.104640.log.gz 2023-08-26T18:17:49.348 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.122873.log 2023-08-26T18:17:49.352 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster1-client.mirror.122873.log: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.122873.log.gz 2023-08-26T18:17:49.353 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.106745.log 2023-08-26T18:17:49.353 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster2-client.admin.106745.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.106745.log.gz 2023-08-26T18:17:49.353 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.113987.log 2023-08-26T18:17:49.355 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster2-client.mirror.113987.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.145428.log 2023-08-26T18:17:49.359 INFO:teuthology.orchestra.run.smithi124.stderr: 93.4% -- replaced with /var/log/ceph/cluster2-client.mirror.113987.log.gz 2023-08-26T18:17:49.369 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster2-client.mirror.145428.log: 8.2% -- replaced with /var/log/ceph/cluster2-client.mirror.145428.log.gz 2023-08-26T18:17:49.369 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.123503.log 2023-08-26T18:17:49.370 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster2-client.mirror.123503.log: 87.4% -- replaced with /var/log/ceph/cluster2-client.mirror.123503.log.gz 2023-08-26T18:17:49.370 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.114477.log 2023-08-26T18:17:49.370 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster2-client.mirror.114477.log: 88.4% -- replaced with /var/log/ceph/cluster2-client.mirror.114477.log.gz 2023-08-26T18:17:49.371 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.138635.log 2023-08-26T18:17:49.371 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster1-client.mirror.138635.log: 86.8% -- replaced with /var/log/ceph/cluster1-client.mirror.138635.log.gz 2023-08-26T18:17:49.371 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.113729.log 2023-08-26T18:17:49.372 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster2-client.mirror.113729.log: 94.1% -- replaced with /var/log/ceph/cluster2-client.mirror.106124.log.gz 2023-08-26T18:17:49.372 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.137308.log 2023-08-26T18:17:49.372 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster1-client.mirror.137308.log: 86.8% -- replaced with /var/log/ceph/cluster1-client.mirror.137308.log.gz 2023-08-26T18:17:49.372 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.109793.log 2023-08-26T18:17:49.373 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster1-client.mirror.109793.log: 93.4% -- replaced with /var/log/ceph/cluster2-client.mirror.113729.log.gz 2023-08-26T18:17:49.373 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.112857.log 2023-08-26T18:17:49.374 INFO:teuthology.orchestra.run.smithi124.stderr: 93.4% -- replaced with /var/log/ceph/cluster1-client.mirror.109793.log.gz 2023-08-26T18:17:49.375 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster2-client.mirror.112857.log: 90.9% -- replaced with /var/log/ceph/cluster2-client.mirror.112857.log.gz 2023-08-26T18:17:49.375 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.115604.log 2023-08-26T18:17:49.376 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster2-client.mirror.115604.log: 79.6% -- replaced with /var/log/ceph/cluster2-client.mirror.115604.log.gz 2023-08-26T18:17:49.376 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.118447.log 2023-08-26T18:17:49.377 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster2-client.mirror.118447.log: 89.7% -- replaced with /var/log/ceph/cluster2-client.mirror.118447.log.gz 2023-08-26T18:17:49.378 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.145314.log 2023-08-26T18:17:49.378 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster2-client.mirror.145314.log: 7.1% -- replaced with /var/log/ceph/cluster2-client.mirror.145314.log.gz 2023-08-26T18:17:49.379 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.130904.log 2023-08-26T18:17:49.380 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster2-client.mirror.130904.log: 87.4% -- replaced with /var/log/ceph/cluster2-client.mirror.130904.log.gz 2023-08-26T18:17:49.380 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.110941.log 2023-08-26T18:17:49.385 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster1-client.mirror.110941.log: 93.4% -- replaced with /var/log/ceph/cluster1-client.mirror.110941.log.gz 2023-08-26T18:17:49.385 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.126416.log 2023-08-26T18:17:49.386 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster1-client.mirror.126416.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.126416.log.gz 2023-08-26T18:17:49.386 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.137938.log 2023-08-26T18:17:49.387 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster1-client.mirror.137938.log: 86.6% -- replaced with /var/log/ceph/cluster1-client.mirror.137938.log.gz 2023-08-26T18:17:49.387 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.143266.log 2023-08-26T18:17:49.389 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster1-client.mirror.143266.log: 2.3% -- replaced with /var/log/ceph/cluster1-client.mirror.143266.log.gz 2023-08-26T18:17:49.389 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.131063.log 2023-08-26T18:17:49.390 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster2-client.mirror.131063.log: 89.7% -- replaced with /var/log/ceph/cluster2-client.mirror.131063.log.gz 2023-08-26T18:17:49.391 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.137983.log 2023-08-26T18:17:49.391 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster1-client.mirror.137983.log: 86.8% -- replaced with /var/log/ceph/cluster1-client.mirror.137983.log.gz 2023-08-26T18:17:49.392 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.138590.log 2023-08-26T18:17:49.392 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster1-client.mirror.138590.log: 86.8% -- replaced with /var/log/ceph/cluster1-client.mirror.138590.log.gz 2023-08-26T18:17:49.393 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.143283.log 2023-08-26T18:17:49.394 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster1-client.admin.143283.log: 1.2% -- replaced with /var/log/ceph/cluster1-client.admin.143283.log.gz 2023-08-26T18:17:49.394 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.132592.log 2023-08-26T18:17:49.395 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster1-client.mirror.132592.log: 92.3% -- replaced with /var/log/ceph/cluster1-client.mirror.132592.log.gz 2023-08-26T18:17:49.396 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.106205.log 2023-08-26T18:17:49.396 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster1-client.mirror.106205.log: 90.3% -- replaced with /var/log/ceph/cluster1-client.mirror.106205.log.gz 2023-08-26T18:17:49.397 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.109275.log 2023-08-26T18:17:49.401 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster1-client.mirror.109275.log: 93.7% -- replaced with /var/log/ceph/cluster2-client.mirror.116653.log.gz 2023-08-26T18:17:49.402 INFO:teuthology.orchestra.run.smithi124.stderr: 93.4% -- replaced with /var/log/ceph/cluster1-client.mirror.109275.log.gz 2023-08-26T18:17:49.402 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.113448.log 2023-08-26T18:17:49.403 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.118800.log 2023-08-26T18:17:49.404 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster1-client.mirror.113448.log: /var/log/ceph/cluster2-client.mirror.118800.log: 89.7% -- replaced with /var/log/ceph/cluster2-client.mirror.118800.log.gz 2023-08-26T18:17:49.404 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.122448.log 2023-08-26T18:17:49.406 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster2-client.mirror.122448.log: 92.5% -- replaced with /var/log/ceph/cluster2-client.mirror.122448.log.gz 2023-08-26T18:17:49.406 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.136685.log 2023-08-26T18:17:49.407 INFO:teuthology.orchestra.run.smithi124.stderr: 93.4% -- replaced with /var/log/ceph/cluster1-client.mirror.113448.log.gz 2023-08-26T18:17:49.408 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.121823.log 2023-08-26T18:17:49.408 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster2-client.mirror.136685.log: /var/log/ceph/cluster2-client.mirror.121823.log: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.136685.log.gz 2023-08-26T18:17:49.409 INFO:teuthology.orchestra.run.smithi124.stderr: 87.4% -- replaced with /var/log/ceph/cluster2-client.mirror.121823.log.gz 2023-08-26T18:17:49.409 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.119089.log 2023-08-26T18:17:49.411 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.131398.log 2023-08-26T18:17:49.411 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster2-client.mirror.119089.log: /var/log/ceph/cluster1-client.mirror.131398.log: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.131398.log.gz 2023-08-26T18:17:49.411 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.106550.log 2023-08-26T18:17:49.413 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster1-client.admin.106550.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.106550.log.gz 2023-08-26T18:17:49.413 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.110441.log 2023-08-26T18:17:49.416 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster2-client.mirror.110441.log: 73.4% -- replaced with /var/log/ceph/cluster2-client.mirror.110441.log.gz 2023-08-26T18:17:49.416 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.119146.log 2023-08-26T18:17:49.417 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster2-client.mirror.119146.log: 87.5% -- replaced with /var/log/ceph/cluster2-client.mirror.119146.log.gz 2023-08-26T18:17:49.417 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.126801.log 2023-08-26T18:17:49.418 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster2-client.mirror.126801.log: 89.7% -- replaced with /var/log/ceph/cluster2-client.mirror.126801.log.gz 2023-08-26T18:17:49.418 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-osd.2.log 2023-08-26T18:17:49.429 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster2-osd.2.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.131291.log 2023-08-26T18:17:49.435 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster1-client.mirror.131291.log: 92.2% -- replaced with /var/log/ceph/cluster1-client.mirror.131291.log.gz 2023-08-26T18:17:49.438 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-osd.1.log 2023-08-26T18:17:49.459 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster2-osd.1.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.2.130432.log 2023-08-26T18:17:49.470 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster1-client.mirror.2.130432.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.118625.log 2023-08-26T18:17:49.475 INFO:teuthology.orchestra.run.smithi124.stderr: 93.7% -- replaced with /var/log/ceph/cluster2-client.mirror.111028.log.gz 2023-08-26T18:17:49.476 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster2-client.mirror.118625.log: 87.4% -- replaced with /var/log/ceph/cluster2-client.mirror.118625.log.gz 2023-08-26T18:17:49.477 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.130976.log 2023-08-26T18:17:49.478 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster1-client.mirror.130976.log: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.130976.log.gz 2023-08-26T18:17:49.478 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.128941.log 2023-08-26T18:17:49.478 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster1-client.mirror.128941.log: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.128941.log.gz 2023-08-26T18:17:49.479 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.113534.log 2023-08-26T18:17:49.496 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster1-client.mirror.113534.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.137803.log 2023-08-26T18:17:49.501 INFO:teuthology.orchestra.run.smithi124.stderr: 93.4% -- replaced with /var/log/ceph/cluster1-client.mirror.113534.log.gz 2023-08-26T18:17:49.513 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster1-client.mirror.137803.log: 86.8% -- replaced with /var/log/ceph/cluster1-client.mirror.137803.log.gz 2023-08-26T18:17:49.514 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.106799.log 2023-08-26T18:17:49.526 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster2-client.admin.106799.log: 80.0% -- replaced with /var/log/ceph/cluster2-client.admin.106799.log.gz 2023-08-26T18:17:49.526 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.126440.log 2023-08-26T18:17:49.534 INFO:teuthology.orchestra.run.smithi124.stderr: 92.2% -- replaced with /var/log/ceph/cluster1-client.mirror.2.130432.log.gz 2023-08-26T18:17:49.535 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster1-client.mirror.126440.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.126440.log.gz 2023-08-26T18:17:49.535 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.138388.log 2023-08-26T18:17:49.536 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster1-client.mirror.138388.log: 86.8% -- replaced with /var/log/ceph/cluster1-client.mirror.138388.log.gz 2023-08-26T18:17:49.536 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.129512.log 2023-08-26T18:17:49.538 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster1-client.mirror.129512.log: 92.3% -- replaced with /var/log/ceph/cluster1-client.mirror.129512.log.gz 2023-08-26T18:17:49.539 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.127840.log 2023-08-26T18:17:49.539 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster1-client.mirror.127840.log: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.127840.log.gz 2023-08-26T18:17:49.540 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.116047.log 2023-08-26T18:17:49.555 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster2-client.mirror.116047.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.112484.log 2023-08-26T18:17:49.564 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster2-client.mirror.112484.log: 90.0% -- replaced with /var/log/ceph/cluster2-client.mirror.112484.log.gz 2023-08-26T18:17:49.564 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.118605.log 2023-08-26T18:17:49.576 INFO:teuthology.orchestra.run.smithi124.stderr: 93.7% -- replaced with /var/log/ceph/cluster2-client.mirror.119605.log.gz 2023-08-26T18:17:49.578 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster2-client.mirror.118605.log: 89.7% -- replaced with /var/log/ceph/cluster2-client.mirror.118605.log.gz 2023-08-26T18:17:49.579 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.105698.log 2023-08-26T18:17:49.579 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster1-client.admin.105698.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.105698.log.gz 2023-08-26T18:17:49.580 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.2.119999.log 2023-08-26T18:17:49.582 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster1-client.mirror.2.119999.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.106991.log 2023-08-26T18:17:49.592 INFO:teuthology.orchestra.run.smithi124.stderr: 93.7% -- replaced with /var/log/ceph/cluster2-client.mirror.114948.log.gz 2023-08-26T18:17:49.594 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster2-client.mirror.106991.log: 89.7% -- replaced with /var/log/ceph/cluster2-client.mirror.106991.log.gz 2023-08-26T18:17:49.595 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.0.117979.log 2023-08-26T18:17:49.605 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster1-client.mirror.0.117979.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.123373.log 2023-08-26T18:17:49.618 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster1-client.mirror.123373.log: 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.123373.log.gz 2023-08-26T18:17:49.619 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.115268.log 2023-08-26T18:17:49.640 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster2-client.mirror.115268.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.135899.log 2023-08-26T18:17:49.654 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster2-client.mirror.135899.log: 88.0% -- replaced with /var/log/ceph/cluster2-client.mirror.135899.log.gz 2023-08-26T18:17:49.655 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.125268.log 2023-08-26T18:17:49.655 INFO:teuthology.orchestra.run.smithi124.stderr: 93.4% -- replaced with /var/log/ceph/cluster1-client.mirror.2.119999.log.gz 2023-08-26T18:17:49.656 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster2-client.mirror.125268.log: 89.6% -- replaced with /var/log/ceph/cluster2-client.mirror.125268.log.gz 2023-08-26T18:17:49.666 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.117517.log 2023-08-26T18:17:49.680 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster2-client.mirror.117517.log: gzip -5 --verbose -- /var/log/ceph/ceph.tmp-client.admin.94635.log 2023-08-26T18:17:49.692 INFO:teuthology.orchestra.run.smithi124.stderr: 93.3% -- replaced with /var/log/ceph/cluster1-client.mirror.0.117979.log.gz 2023-08-26T18:17:49.692 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/ceph.tmp-client.admin.94635.log: 0.0% -- replaced with /var/log/ceph/ceph.tmp-client.admin.94635.log.gz 2023-08-26T18:17:49.692 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.112176.log 2023-08-26T18:17:49.692 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster2-client.mirror.112176.log: 91.7% -- replaced with /var/log/ceph/cluster2-client.mirror.112176.log.gz 2023-08-26T18:17:49.693 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.127186.log 2023-08-26T18:17:49.693 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster1-client.mirror.127186.log: 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.127186.log.gz 2023-08-26T18:17:49.693 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.116005.log 2023-08-26T18:17:49.694 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster2-client.mirror.116005.log: 89.7% -- replaced with /var/log/ceph/cluster2-client.mirror.116005.log.gz 2023-08-26T18:17:49.696 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.135465.log 2023-08-26T18:17:49.697 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster2-client.mirror.135465.log: 88.0% -- replaced with /var/log/ceph/cluster2-client.mirror.135465.log.gz 2023-08-26T18:17:49.697 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.132455.log 2023-08-26T18:17:49.697 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster2-client.mirror.132455.log: 87.3% -- replaced with /var/log/ceph/cluster2-client.mirror.132455.log.gz 2023-08-26T18:17:49.697 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.106626.log 2023-08-26T18:17:49.698 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster1-client.mirror.106626.log: 53.5% -- replaced with /var/log/ceph/cluster1-client.mirror.106626.log.gz 2023-08-26T18:17:49.698 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.113620.log 2023-08-26T18:17:49.701 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster1-client.mirror.113620.log: 93.4% -- replaced with /var/log/ceph/cluster1-client.mirror.113620.log.gz 2023-08-26T18:17:49.701 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.124810.log 2023-08-26T18:17:49.702 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster1-client.mirror.124810.log: 94.9% -- replaced with /var/log/ceph/cluster1-client.mirror.124810.log.gz 2023-08-26T18:17:49.702 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.142544.log 2023-08-26T18:17:49.702 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster1-client.mirror.142544.log: 86.8% -- replaced with /var/log/ceph/cluster1-client.mirror.142544.log.gz 2023-08-26T18:17:49.702 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.136903.log 2023-08-26T18:17:49.703 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster1-client.mirror.136903.log: 86.8% -- replaced with /var/log/ceph/cluster1-client.mirror.136903.log.gz 2023-08-26T18:17:49.703 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.143228.log 2023-08-26T18:17:49.705 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster1-client.mirror.143228.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.143228.log.gz 2023-08-26T18:17:49.705 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.133054.log 2023-08-26T18:17:49.705 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster2-client.mirror.133054.log: 93.7% -- replaced with /var/log/ceph/cluster2-client.mirror.119089.log.gz 2023-08-26T18:17:49.706 INFO:teuthology.orchestra.run.smithi124.stderr: 89.7% -- replaced with /var/log/ceph/cluster2-client.mirror.133054.log.gz 2023-08-26T18:17:49.706 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.141197.log 2023-08-26T18:17:49.706 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.134367.log 2023-08-26T18:17:49.707 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster1-client.mirror.141197.log: /var/log/ceph/cluster2-client.mirror.134367.log: 87.2% -- replaced with /var/log/ceph/cluster1-client.mirror.141197.log.gz 2023-08-26T18:17:49.707 INFO:teuthology.orchestra.run.smithi124.stderr: 88.0% -- replaced with /var/log/ceph/cluster2-client.mirror.134367.log.gz 2023-08-26T18:17:49.707 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.122801.log 2023-08-26T18:17:49.708 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster1-client.mirror.122801.log: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.122801.log.gz 2023-08-26T18:17:49.708 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.123117.log 2023-08-26T18:17:49.710 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster1-client.mirror.123117.log: 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.123117.log.gz 2023-08-26T18:17:49.710 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.113271.log 2023-08-26T18:17:49.710 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster2-client.mirror.113271.log: 90.0% -- replaced with /var/log/ceph/cluster2-client.mirror.113271.log.gz 2023-08-26T18:17:49.711 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.120381.log 2023-08-26T18:17:49.711 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster1-client.mirror.120381.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.120381.log.gz 2023-08-26T18:17:49.712 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.115314.log 2023-08-26T18:17:49.712 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster2-client.mirror.115314.log: 89.7% -- replaced with /var/log/ceph/cluster2-client.mirror.115314.log.gz 2023-08-26T18:17:49.712 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.140858.log 2023-08-26T18:17:49.714 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster2-client.mirror.140858.log: 88.7% -- replaced with /var/log/ceph/cluster2-client.mirror.140858.log.gz 2023-08-26T18:17:49.714 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.114437.log 2023-08-26T18:17:49.714 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster2-client.mirror.114437.log: 88.5% -- replaced with /var/log/ceph/cluster2-client.mirror.114437.log.gz 2023-08-26T18:17:49.714 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.143908.log 2023-08-26T18:17:49.715 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster2-client.mirror.143908.log: 8.2% -- replaced with /var/log/ceph/cluster2-client.mirror.143908.log.gz 2023-08-26T18:17:49.715 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.124529.log 2023-08-26T18:17:49.715 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster1-client.mirror.124529.log: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.124529.log.gz 2023-08-26T18:17:49.716 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.122938.log 2023-08-26T18:17:49.716 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster1-client.mirror.122938.log: 92.2% -- replaced with /var/log/ceph/cluster1-client.mirror.122938.log.gz 2023-08-26T18:17:49.717 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.140294.log 2023-08-26T18:17:49.717 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster1-client.mirror.140294.log: 87.2% -- replaced with /var/log/ceph/cluster1-client.mirror.140294.log.gz 2023-08-26T18:17:49.718 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.109728.log 2023-08-26T18:17:49.719 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster1-client.mirror.109728.log: 88.0% -- replaced with /var/log/ceph/cluster1-client.mirror.109728.log.gz 2023-08-26T18:17:49.719 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.114697.log 2023-08-26T18:17:49.719 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster2-client.mirror.114697.log: 88.1% -- replaced with /var/log/ceph/cluster2-client.mirror.114697.log.gz 2023-08-26T18:17:49.719 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.121262.log 2023-08-26T18:17:49.721 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster2-client.mirror.121262.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.119393.log 2023-08-26T18:17:49.724 INFO:teuthology.orchestra.run.smithi124.stderr: 89.8% -- replaced with /var/log/ceph/cluster2-client.mirror.121262.log.gz 2023-08-26T18:17:49.725 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster2-client.mirror.119393.log: 87.5% -- replaced with /var/log/ceph/cluster2-client.mirror.119393.log.gz 2023-08-26T18:17:49.725 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.109394.log 2023-08-26T18:17:49.729 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster1-client.admin.109394.log: 87.7% -- replaced with /var/log/ceph/cluster1-client.admin.109394.log.gz 2023-08-26T18:17:49.729 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.129655.log 2023-08-26T18:17:49.729 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster1-client.mirror.129655.log: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.129655.log.gz 2023-08-26T18:17:49.730 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.111818.log 2023-08-26T18:17:49.730 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster2-client.mirror.111818.log: 88.9% -- replaced with /var/log/ceph/cluster2-client.mirror.111818.log.gz 2023-08-26T18:17:49.731 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.138928.log 2023-08-26T18:17:49.731 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster1-client.mirror.138928.log: 86.8% -- replaced with /var/log/ceph/cluster1-client.mirror.138928.log.gz 2023-08-26T18:17:49.731 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.136948.log 2023-08-26T18:17:49.732 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster1-client.mirror.136948.log: 86.8% -- replaced with /var/log/ceph/cluster1-client.mirror.136948.log.gz 2023-08-26T18:17:49.732 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.137263.log 2023-08-26T18:17:49.733 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster1-client.mirror.137263.log: 86.8% -- replaced with /var/log/ceph/cluster1-client.mirror.137263.log.gz 2023-08-26T18:17:49.733 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.126509.log 2023-08-26T18:17:49.733 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster2-client.mirror.126509.log: 92.4% -- replaced with /var/log/ceph/cluster2-client.mirror.126509.log.gz 2023-08-26T18:17:49.734 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.120198.log 2023-08-26T18:17:49.735 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster2-client.mirror.120198.log: 1.3% -- replaced with /var/log/ceph/cluster2-client.mirror.120198.log.gz 2023-08-26T18:17:49.735 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.114159.log 2023-08-26T18:17:49.738 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster2-client.mirror.114159.log: 93.4% -- replaced with /var/log/ceph/cluster2-client.mirror.114159.log.gz 2023-08-26T18:17:49.738 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.120595.log 2023-08-26T18:17:49.738 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster2-client.mirror.120595.log: 89.7% -- replaced with /var/log/ceph/cluster2-client.mirror.120595.log.gz 2023-08-26T18:17:49.739 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.127082.log 2023-08-26T18:17:49.748 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster2-client.mirror.127082.log: 87.3% -- replaced with /var/log/ceph/cluster2-client.mirror.127082.log.gz 2023-08-26T18:17:49.749 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.146156.log 2023-08-26T18:17:49.749 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster2-client.admin.146156.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.146156.log.gz 2023-08-26T18:17:49.749 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.137195.log 2023-08-26T18:17:49.750 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster1-client.mirror.137195.log: 85.0% -- replaced with /var/log/ceph/cluster1-client.mirror.137195.log.gz 2023-08-26T18:17:49.750 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.121103.log 2023-08-26T18:17:49.751 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster2-client.mirror.121103.log: 87.4% -- replaced with /var/log/ceph/cluster2-client.mirror.121103.log.gz 2023-08-26T18:17:49.753 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.139040.log 2023-08-26T18:17:49.763 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster1-client.mirror.139040.log: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.139040.log.gz 2023-08-26T18:17:49.764 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.110691.log 2023-08-26T18:17:49.764 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster1-client.mirror.110691.log: 73.5% -- replaced with /var/log/ceph/cluster1-client.mirror.110691.log.gz 2023-08-26T18:17:49.764 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.131574.log 2023-08-26T18:17:49.765 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster1-client.mirror.131574.log: 94.4% -- replaced with /var/log/ceph/cluster1-client.mirror.131574.log.gz 2023-08-26T18:17:49.766 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.113382.log 2023-08-26T18:17:49.766 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster1-client.mirror.113382.log: 89.3% -- replaced with /var/log/ceph/cluster1-client.mirror.113382.log.gz 2023-08-26T18:17:49.766 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.137083.log 2023-08-26T18:17:49.767 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster1-client.mirror.137083.log: 86.8% -- replaced with /var/log/ceph/cluster1-client.mirror.137083.log.gz 2023-08-26T18:17:49.767 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.125347.log 2023-08-26T18:17:49.768 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster2-client.mirror.125347.log: 87.3% -- replaced with /var/log/ceph/cluster2-client.mirror.125347.log.gz 2023-08-26T18:17:49.768 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.105744.log 2023-08-26T18:17:49.769 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster2-client.admin.105744.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.105744.log.gz 2023-08-26T18:17:49.769 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.118951.log 2023-08-26T18:17:49.769 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster2-client.mirror.118951.log: 79.6% -- replaced with /var/log/ceph/cluster2-client.mirror.118951.log.gz 2023-08-26T18:17:49.770 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.135754.log 2023-08-26T18:17:49.774 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster2-client.mirror.135754.log: 90.4% -- replaced with /var/log/ceph/cluster2-client.mirror.135754.log.gz 2023-08-26T18:17:49.774 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.130050.log 2023-08-26T18:17:49.775 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster1-client.mirror.130050.log: 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.130050.log.gz 2023-08-26T18:17:49.775 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.113663.log 2023-08-26T18:17:49.780 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster1-client.mirror.113663.log: 93.4% -- replaced with /var/log/ceph/cluster1-client.mirror.113663.log.gz 2023-08-26T18:17:49.780 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.119684.log 2023-08-26T18:17:49.787 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster2-client.mirror.119684.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.119284.log 2023-08-26T18:17:49.799 INFO:teuthology.orchestra.run.smithi124.stderr: 93.7% -- replaced with /var/log/ceph/cluster2-client.mirror.116047.log.gz 2023-08-26T18:17:49.800 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster2-client.mirror.119284.log: 89.7% -- replaced with /var/log/ceph/cluster2-client.mirror.119284.log.gz 2023-08-26T18:17:49.814 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.123233.log 2023-08-26T18:17:49.814 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster2-client.mirror.123233.log: 89.8% -- replaced with /var/log/ceph/cluster2-client.mirror.123233.log.gz 2023-08-26T18:17:49.814 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.131715.log 2023-08-26T18:17:49.815 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster2-client.mirror.131715.log: 87.4% -- replaced with /var/log/ceph/cluster2-client.mirror.131715.log.gz 2023-08-26T18:17:49.815 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.126759.log 2023-08-26T18:17:49.815 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster2-client.mirror.126759.log: 92.5% -- replaced with /var/log/ceph/cluster2-client.mirror.126759.log.gz 2023-08-26T18:17:49.816 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.137398.log 2023-08-26T18:17:49.816 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster1-client.mirror.137398.log: 86.7% -- replaced with /var/log/ceph/cluster1-client.mirror.137398.log.gz 2023-08-26T18:17:49.816 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.107957.log 2023-08-26T18:17:49.816 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster2-client.mirror.107957.log: 89.8% -- replaced with /var/log/ceph/cluster2-client.mirror.107957.log.gz 2023-08-26T18:17:49.817 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.99782.log 2023-08-26T18:17:49.817 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster2-client.admin.99782.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.99782.log.gz 2023-08-26T18:17:49.817 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.114030.log 2023-08-26T18:17:49.821 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster2-client.mirror.114030.log: 93.5% -- replaced with /var/log/ceph/cluster2-client.mirror.114030.log.gz 2023-08-26T18:17:49.822 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.113471.log 2023-08-26T18:17:49.822 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster2-client.mirror.113471.log: 93.4% -- replaced with /var/log/ceph/cluster2-client.mirror.113471.log.gz 2023-08-26T18:17:49.822 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.129587.log 2023-08-26T18:17:49.822 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster2-client.mirror.129587.log: 89.8% -- replaced with /var/log/ceph/cluster2-client.mirror.129587.log.gz 2023-08-26T18:17:49.823 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.140837.log 2023-08-26T18:17:49.824 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster1-client.mirror.140837.log: 87.2% -- replaced with /var/log/ceph/cluster1-client.mirror.140837.log.gz 2023-08-26T18:17:49.824 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.111418.log 2023-08-26T18:17:49.824 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster1-client.mirror.111418.log: 88.2% -- replaced with /var/log/ceph/cluster1-client.mirror.111418.log.gz 2023-08-26T18:17:49.825 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.109912.log 2023-08-26T18:17:49.825 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster1-client.admin.109912.log: 87.5% -- replaced with /var/log/ceph/cluster1-client.admin.109912.log.gz 2023-08-26T18:17:49.825 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.135298.log 2023-08-26T18:17:49.826 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster2-client.mirror.135298.log: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.135298.log.gz 2023-08-26T18:17:49.826 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.132895.log 2023-08-26T18:17:49.826 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster2-client.mirror.132895.log: 87.4% -- replaced with /var/log/ceph/cluster2-client.mirror.132895.log.gz 2023-08-26T18:17:49.827 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.111979.log 2023-08-26T18:17:49.832 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster1-client.mirror.111979.log: 93.3% -- replaced with /var/log/ceph/cluster1-client.mirror.111979.log.gz 2023-08-26T18:17:49.832 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.133584.log 2023-08-26T18:17:49.832 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster1-client.mirror.133584.log: 87.6% -- replaced with /var/log/ceph/cluster1-client.mirror.133584.log.gz 2023-08-26T18:17:49.833 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.130166.log 2023-08-26T18:17:49.833 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster2-client.mirror.130166.log: 89.9% -- replaced with /var/log/ceph/cluster2-client.mirror.130166.log.gz 2023-08-26T18:17:49.834 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.123483.log 2023-08-26T18:17:49.835 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster2-client.mirror.123483.log: 89.8% -- replaced with /var/log/ceph/cluster2-client.mirror.123483.log.gz 2023-08-26T18:17:49.835 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.140814.log 2023-08-26T18:17:49.835 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster1-client.mirror.140814.log: 87.2% -- replaced with /var/log/ceph/cluster1-client.mirror.140814.log.gz 2023-08-26T18:17:49.836 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.142340.log 2023-08-26T18:17:49.836 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster1-client.mirror.142340.log: 93.7% -- replaced with /var/log/ceph/cluster2-client.mirror.141899.log.gz 2023-08-26T18:17:49.836 INFO:teuthology.orchestra.run.smithi124.stderr: 86.8% -- replaced with /var/log/ceph/cluster1-client.mirror.142340.log.gz 2023-08-26T18:17:49.837 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.135589.log 2023-08-26T18:17:49.838 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.121524.log 2023-08-26T18:17:49.839 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster2-client.mirror.135589.log: /var/log/ceph/cluster2-client.mirror.121524.log: 88.0% -- replaced with /var/log/ceph/cluster2-client.mirror.135589.log.gz 2023-08-26T18:17:49.839 INFO:teuthology.orchestra.run.smithi124.stderr: 89.8% -- replaced with /var/log/ceph/cluster2-client.mirror.121524.log.gz 2023-08-26T18:17:49.839 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.143360.log 2023-08-26T18:17:49.840 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster2-client.admin.143360.log: 8.2% -- replaced with /var/log/ceph/cluster2-client.admin.143360.log.gz 2023-08-26T18:17:49.840 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.115455.log 2023-08-26T18:17:49.847 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster2-client.mirror.115455.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.116495.log 2023-08-26T18:17:49.851 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster2-client.mirror.116495.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.126158.log 2023-08-26T18:17:49.874 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster2-client.mirror.126158.log: 87.4% -- replaced with /var/log/ceph/cluster2-client.mirror.126158.log.gz 2023-08-26T18:17:49.874 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.107882.log 2023-08-26T18:17:49.881 INFO:teuthology.orchestra.run.smithi124.stderr: 93.7% -- replaced with /var/log/ceph/cluster2-client.mirror.115268.log.gz 2023-08-26T18:17:49.882 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster1-client.mirror.107882.log: 87.1% -- replaced with /var/log/ceph/cluster1-client.mirror.107882.log.gz 2023-08-26T18:17:49.883 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.133101.log 2023-08-26T18:17:49.883 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster2-client.mirror.133101.log: 89.8% -- replaced with /var/log/ceph/cluster2-client.mirror.133101.log.gz 2023-08-26T18:17:49.884 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.119999.log 2023-08-26T18:17:49.895 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster2-client.mirror.119999.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.115988.log 2023-08-26T18:17:49.920 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster2-client.mirror.115988.log: 79.5% -- replaced with /var/log/ceph/cluster2-client.mirror.115988.log.gz 2023-08-26T18:17:49.921 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.118588.log 2023-08-26T18:17:49.924 INFO:teuthology.orchestra.run.smithi124.stderr: 93.5% -- replaced with /var/log/ceph/cluster2-client.mirror.119999.log.gz 2023-08-26T18:17:49.925 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster2-client.mirror.118588.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.136375.log 2023-08-26T18:17:49.925 INFO:teuthology.orchestra.run.smithi124.stderr: 79.4% -- replaced with /var/log/ceph/cluster2-client.mirror.118588.log.gz 2023-08-26T18:17:49.926 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster2-client.mirror.136375.log: 88.1% -- replaced with /var/log/ceph/cluster2-client.mirror.136375.log.gz 2023-08-26T18:17:49.927 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.121571.log 2023-08-26T18:17:49.928 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster2-client.mirror.121571.log: 89.7% -- replaced with /var/log/ceph/cluster2-client.mirror.121571.log.gz 2023-08-26T18:17:49.928 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.130234.log 2023-08-26T18:17:49.929 INFO:teuthology.orchestra.run.smithi031.stderr: 90.2% -- replaced with /var/log/ceph/cluster1-mon.a.log.gz 2023-08-26T18:17:49.930 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster1-client.mirror.130234.log: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.130234.log.gz 2023-08-26T18:17:49.930 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.142763.log 2023-08-26T18:17:49.931 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster1-client.mirror.142763.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.142763.log.gz 2023-08-26T18:17:49.931 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.131154.log 2023-08-26T18:17:49.932 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster1-client.mirror.131154.log: 87.2% -- replaced with /var/log/ceph/cluster1-client.mirror.131154.log.gz 2023-08-26T18:17:49.932 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.136072.log 2023-08-26T18:17:49.934 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster2-client.mirror.136072.log: 90.4% -- replaced with /var/log/ceph/cluster2-client.mirror.136072.log.gz 2023-08-26T18:17:49.935 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.121964.log 2023-08-26T18:17:49.937 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster2-client.mirror.121964.log: 92.4% -- replaced with /var/log/ceph/cluster2-client.mirror.121964.log.gz 2023-08-26T18:17:49.937 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.117819.log 2023-08-26T18:17:49.938 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster2-client.mirror.117819.log: 87.4% -- replaced with /var/log/ceph/cluster2-client.mirror.117819.log.gz 2023-08-26T18:17:49.938 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.99694.log 2023-08-26T18:17:49.939 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster2-client.admin.99694.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.99694.log.gz 2023-08-26T18:17:49.939 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.114222.log 2023-08-26T18:17:49.945 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster1-client.mirror.114222.log: 93.4% -- replaced with /var/log/ceph/cluster1-client.mirror.114222.log.gz 2023-08-26T18:17:49.945 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.130432.log 2023-08-26T18:17:49.949 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster2-client.mirror.130432.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.124899.log 2023-08-26T18:17:49.960 INFO:teuthology.orchestra.run.smithi124.stderr: 93.3% -- replaced with /var/log/ceph/cluster2-client.mirror.130432.log.gz 2023-08-26T18:17:49.963 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster1-client.mirror.124899.log: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.124899.log.gz 2023-08-26T18:17:49.964 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.119662.log 2023-08-26T18:17:49.964 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster2-client.mirror.119662.log: 87.5% -- replaced with /var/log/ceph/cluster2-client.mirror.119662.log.gz 2023-08-26T18:17:49.964 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.125506.log 2023-08-26T18:17:49.965 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster2-client.mirror.125506.log: 89.8% -- replaced with /var/log/ceph/cluster2-client.mirror.125506.log.gz 2023-08-26T18:17:49.965 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.122897.log 2023-08-26T18:17:49.966 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster1-client.mirror.122897.log: 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.122897.log.gz 2023-08-26T18:17:49.966 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.112274.log 2023-08-26T18:17:49.966 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster1-client.mirror.112274.log: 89.1% -- replaced with /var/log/ceph/cluster1-client.mirror.112274.log.gz 2023-08-26T18:17:49.967 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.111082.log 2023-08-26T18:17:49.968 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster1-client.mirror.111082.log: 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.111082.log.gz 2023-08-26T18:17:49.968 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.124505.log 2023-08-26T18:17:49.969 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster1-client.mirror.124505.log: 87.2% -- replaced with /var/log/ceph/cluster1-client.mirror.124505.log.gz 2023-08-26T18:17:49.969 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.136478.log 2023-08-26T18:17:49.970 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster2-client.mirror.136478.log: 88.0% -- replaced with /var/log/ceph/cluster2-client.mirror.136478.log.gz 2023-08-26T18:17:49.970 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.134884.log 2023-08-26T18:17:49.972 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster2-client.mirror.134884.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.125603.log 2023-08-26T18:17:49.973 INFO:teuthology.orchestra.run.smithi124.stderr: 90.4% -- replaced with /var/log/ceph/cluster2-client.mirror.134884.log.gz 2023-08-26T18:17:49.974 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster1-client.mirror.125603.log: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.125603.log.gz 2023-08-26T18:17:49.975 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.118713.log 2023-08-26T18:17:49.976 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster2-client.mirror.118713.log: 89.7% -- replaced with /var/log/ceph/cluster2-client.mirror.118713.log.gz 2023-08-26T18:17:49.977 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.131130.log 2023-08-26T18:17:49.978 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster2-client.mirror.131130.log: 87.4% -- replaced with /var/log/ceph/cluster2-client.mirror.131130.log.gz 2023-08-26T18:17:49.978 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.134781.log 2023-08-26T18:17:49.979 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster2-client.mirror.134781.log: 88.0% -- replaced with /var/log/ceph/cluster2-client.mirror.134781.log.gz 2023-08-26T18:17:49.979 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.136726.log 2023-08-26T18:17:49.980 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster2-client.mirror.136726.log: 88.0% -- replaced with /var/log/ceph/cluster2-client.mirror.136726.log.gz 2023-08-26T18:17:49.980 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.134387.log 2023-08-26T18:17:49.983 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster2-client.mirror.134387.log: 90.4% -- replaced with /var/log/ceph/cluster2-client.mirror.134387.log.gz 2023-08-26T18:17:49.983 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.127106.log 2023-08-26T18:17:49.984 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster1-client.mirror.127106.log: 87.2% -- replaced with /var/log/ceph/cluster1-client.mirror.127106.log.gz 2023-08-26T18:17:49.984 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.131927.log 2023-08-26T18:17:49.986 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster2-client.mirror.131927.log: 89.8% -- replaced with /var/log/ceph/cluster2-client.mirror.131927.log.gz 2023-08-26T18:17:49.986 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.130680.log 2023-08-26T18:17:49.987 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster2-client.mirror.130680.log: 89.7% -- replaced with /var/log/ceph/cluster2-client.mirror.130680.log.gz 2023-08-26T18:17:49.988 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.110270.log 2023-08-26T18:17:49.988 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster1-client.mirror.110270.log: 88.1% -- replaced with /var/log/ceph/cluster1-client.mirror.110270.log.gz 2023-08-26T18:17:49.989 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.118899.log 2023-08-26T18:17:49.990 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster2-client.mirror.118899.log: 87.4% -- replaced with /var/log/ceph/cluster2-client.mirror.118899.log.gz 2023-08-26T18:17:49.990 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.117438.log 2023-08-26T18:17:50.001 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster2-client.mirror.117438.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.108688.log 2023-08-26T18:17:50.026 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster2-client.mirror.108688.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.141412.log 2023-08-26T18:17:50.047 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster2-client.mirror.141412.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.143061.log 2023-08-26T18:17:50.060 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster1-client.admin.143061.log: 50.4% -- replaced with /var/log/ceph/cluster1-client.admin.143061.log.gz 2023-08-26T18:17:50.071 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.123640.log 2023-08-26T18:17:50.071 INFO:teuthology.orchestra.run.smithi124.stderr: 93.7% -- replaced with /var/log/ceph/cluster2-client.mirror.116495.log.gz 2023-08-26T18:17:50.094 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster1-client.mirror.123640.log: 92.2% -- replaced with /var/log/ceph/cluster1-client.mirror.123640.log.gz 2023-08-26T18:17:50.094 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-mon.a.log 2023-08-26T18:17:50.116 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster2-mon.a.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.109114.log 2023-08-26T18:17:50.119 INFO:teuthology.orchestra.run.smithi124.stderr: 93.7% -- replaced with /var/log/ceph/cluster2-client.mirror.119684.log.gz 2023-08-26T18:17:50.120 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster2-client.mirror.109114.log: 87.8% -- replaced with /var/log/ceph/cluster2-client.mirror.109114.log.gz 2023-08-26T18:17:50.120 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.137623.log 2023-08-26T18:17:50.121 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster1-client.mirror.137623.log: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.137623.log.gz 2023-08-26T18:17:50.121 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.125205.log 2023-08-26T18:17:50.144 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster1-client.mirror.125205.log: 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.125205.log.gz 2023-08-26T18:17:50.144 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.136835.log 2023-08-26T18:17:50.146 INFO:teuthology.orchestra.run.smithi124.stderr: 93.7% -- replaced with /var/log/ceph/cluster2-client.mirror.117517.log.gz 2023-08-26T18:17:50.156 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster1-client.mirror.136835.log: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.136835.log.gz 2023-08-26T18:17:50.157 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.137713.log 2023-08-26T18:17:50.157 INFO:teuthology.orchestra.run.smithi124.stderr: 93.7% -- replaced with /var/log/ceph/cluster2-client.mirror.115455.log.gz 2023-08-26T18:17:50.158 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster1-client.mirror.137713.log: 86.5% -- replaced with /var/log/ceph/cluster1-client.mirror.137713.log.gz 2023-08-26T18:17:50.158 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.139087.log 2023-08-26T18:17:50.159 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster1-client.mirror.139087.log: 1.3% -- replaced with /var/log/ceph/cluster1-client.mirror.139087.log.gz 2023-08-26T18:17:50.159 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.135527.log 2023-08-26T18:17:50.160 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster2-client.mirror.135527.log: 88.1% -- replaced with /var/log/ceph/cluster2-client.mirror.135527.log.gz 2023-08-26T18:17:50.160 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.142409.log 2023-08-26T18:17:50.161 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster1-client.mirror.142409.log: 86.6% -- replaced with /var/log/ceph/cluster1-client.mirror.142409.log.gz 2023-08-26T18:17:50.161 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.115663.log 2023-08-26T18:17:50.172 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster2-client.mirror.115663.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.113901.log 2023-08-26T18:17:50.194 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster2-client.mirror.113901.log: 93.4% -- replaced with /var/log/ceph/cluster2-client.mirror.113901.log.gz 2023-08-26T18:17:50.194 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.106575.log 2023-08-26T18:17:50.207 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster1-client.admin.106575.log: 79.9% -- replaced with /var/log/ceph/cluster1-client.admin.106575.log.gz 2023-08-26T18:17:50.207 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.106816.log 2023-08-26T18:17:50.219 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster2-client.admin.106816.log: 80.0% -- replaced with /var/log/ceph/cluster2-client.admin.106816.log.gz 2023-08-26T18:17:50.219 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.127412.log 2023-08-26T18:17:50.232 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster1-client.mirror.127412.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.127412.log.gz 2023-08-26T18:17:50.232 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.116084.log 2023-08-26T18:17:50.237 INFO:teuthology.orchestra.run.smithi124.stderr: 93.7% -- replaced with /var/log/ceph/cluster2-client.mirror.117438.log.gz 2023-08-26T18:17:50.239 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster2-client.mirror.116084.log: 89.6% -- replaced with /var/log/ceph/cluster2-client.mirror.116084.log.gz 2023-08-26T18:17:50.239 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.119472.log 2023-08-26T18:17:50.240 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster2-client.mirror.119472.log: 87.3% -- replaced with /var/log/ceph/cluster2-client.mirror.119472.log.gz 2023-08-26T18:17:50.240 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.105790.log 2023-08-26T18:17:50.241 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster2-client.admin.105790.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.105790.log.gz 2023-08-26T18:17:50.241 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.135858.log 2023-08-26T18:17:50.242 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster2-client.mirror.135858.log: 88.1% -- replaced with /var/log/ceph/cluster2-client.mirror.135858.log.gz 2023-08-26T18:17:50.242 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.140554.log 2023-08-26T18:17:50.243 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster1-client.mirror.140554.log: 87.2% -- replaced with /var/log/ceph/cluster1-client.mirror.140554.log.gz 2023-08-26T18:17:50.243 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.118467.log 2023-08-26T18:17:50.244 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster2-client.mirror.118467.log: 87.4% -- replaced with /var/log/ceph/cluster2-client.mirror.118467.log.gz 2023-08-26T18:17:50.244 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.121365.log 2023-08-26T18:17:50.245 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster1-client.mirror.121365.log: 87.2% -- replaced with /var/log/ceph/cluster1-client.mirror.121365.log.gz 2023-08-26T18:17:50.245 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.123876.log 2023-08-26T18:17:50.247 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster2-client.mirror.123876.log: 92.6% -- replaced with /var/log/ceph/cluster2-client.mirror.123876.log.gz 2023-08-26T18:17:50.248 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.144899.log 2023-08-26T18:17:50.248 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster2-client.mirror.144899.log: 7.1% -- replaced with /var/log/ceph/cluster2-client.mirror.144899.log.gz 2023-08-26T18:17:50.249 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.141593.log 2023-08-26T18:17:50.250 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster2-client.mirror.141593.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.139181.log 2023-08-26T18:17:50.252 INFO:teuthology.orchestra.run.smithi124.stderr: 91.6% -- replaced with /var/log/ceph/cluster2-client.mirror.141593.log.gz 2023-08-26T18:17:50.257 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster2-client.mirror.139181.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.115433.log 2023-08-26T18:17:50.278 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster2-client.mirror.115433.log: 87.4% -- replaced with /var/log/ceph/cluster2-client.mirror.115433.log.gz 2023-08-26T18:17:50.278 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.135318.log 2023-08-26T18:17:50.289 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster2-client.mirror.135318.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.138343.log 2023-08-26T18:17:50.289 INFO:teuthology.orchestra.run.smithi124.stderr: 90.5% -- replaced with /var/log/ceph/cluster2-client.mirror.135318.log.gz 2023-08-26T18:17:50.300 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster1-client.mirror.138343.log: 86.7% -- replaced with /var/log/ceph/cluster1-client.mirror.138343.log.gz 2023-08-26T18:17:50.300 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.118192.log 2023-08-26T18:17:50.311 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster2-client.mirror.118192.log: 87.4% -- replaced with /var/log/ceph/cluster2-client.mirror.118192.log.gz 2023-08-26T18:17:50.311 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.129934.log 2023-08-26T18:17:50.322 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster2-client.mirror.129934.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.114093.log 2023-08-26T18:17:50.322 INFO:teuthology.orchestra.run.smithi124.stderr: 89.8% -- replaced with /var/log/ceph/cluster2-client.mirror.129934.log.gz 2023-08-26T18:17:50.346 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster1-client.mirror.114093.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.109836.log 2023-08-26T18:17:50.356 INFO:teuthology.orchestra.run.smithi124.stderr: 93.4% -- replaced with /var/log/ceph/cluster1-client.mirror.114093.log.gz 2023-08-26T18:17:50.356 INFO:teuthology.orchestra.run.smithi124.stderr: 93.7% -- replaced with /var/log/ceph/cluster2-client.mirror.108688.log.gz 2023-08-26T18:17:50.367 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster1-client.mirror.109836.log: 93.4% -- replaced with /var/log/ceph/cluster1-client.mirror.109836.log.gz 2023-08-26T18:17:50.367 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.133983.log 2023-08-26T18:17:50.367 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster2-client.mirror.133983.log: 88.1% -- replaced with /var/log/ceph/cluster2-client.mirror.133983.log.gz 2023-08-26T18:17:50.368 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.137173.log 2023-08-26T18:17:50.368 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster1-client.mirror.137173.log: 86.6% -- replaced with /var/log/ceph/cluster1-client.mirror.137173.log.gz 2023-08-26T18:17:50.368 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.104732.log 2023-08-26T18:17:50.369 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster2-client.admin.104732.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.104732.log.gz 2023-08-26T18:17:50.369 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.137443.log 2023-08-26T18:17:50.369 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster1-client.mirror.137443.log: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.137443.log.gz 2023-08-26T18:17:50.369 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.115570.log 2023-08-26T18:17:50.380 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster2-client.mirror.115570.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.127162.log 2023-08-26T18:17:50.380 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster1-client.mirror.127162.log: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.127162.log.gz 2023-08-26T18:17:50.380 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.138748.log 2023-08-26T18:17:50.389 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster1-client.mirror.138748.log: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.138748.log.gz 2023-08-26T18:17:50.389 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.121919.log 2023-08-26T18:17:50.411 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster1-client.mirror.121919.log: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.121919.log.gz 2023-08-26T18:17:50.411 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.137285.log 2023-08-26T18:17:50.423 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster1-client.mirror.137285.log: 86.7% -- replaced with /var/log/ceph/cluster1-client.mirror.137285.log.gz 2023-08-26T18:17:50.433 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.132812.log 2023-08-26T18:17:50.449 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster1-client.mirror.132812.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.135568.log 2023-08-26T18:17:50.459 INFO:teuthology.orchestra.run.smithi124.stderr: 92.3% -- replaced with /var/log/ceph/cluster1-client.mirror.132812.log.gz 2023-08-26T18:17:50.464 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster2-client.mirror.135568.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.132875.log 2023-08-26T18:17:50.465 INFO:teuthology.orchestra.run.smithi124.stderr: 90.4% -- replaced with /var/log/ceph/cluster2-client.mirror.135568.log.gz 2023-08-26T18:17:50.476 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster2-client.mirror.132875.log: 89.7% -- replaced with /var/log/ceph/cluster2-client.mirror.132875.log.gz 2023-08-26T18:17:50.476 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.105284.log 2023-08-26T18:17:50.487 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster2-client.admin.105284.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.105284.log.gz 2023-08-26T18:17:50.498 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.113964.log 2023-08-26T18:17:50.520 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster1-client.mirror.113964.log: 93.3% -- replaced with /var/log/ceph/cluster1-client.mirror.113964.log.gz 2023-08-26T18:17:50.521 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.113514.log 2023-08-26T18:17:50.537 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster2-client.mirror.113514.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.114778.log 2023-08-26T18:17:50.549 INFO:teuthology.orchestra.run.smithi124.stderr: 93.4% -- replaced with /var/log/ceph/cluster2-client.mirror.113514.log.gz 2023-08-26T18:17:50.560 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster2-client.mirror.114778.log: 88.0% -- replaced with /var/log/ceph/cluster2-client.mirror.114778.log.gz 2023-08-26T18:17:50.560 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.135277.log 2023-08-26T18:17:50.571 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster2-client.mirror.135277.log: 88.0% -- replaced with /var/log/ceph/cluster2-client.mirror.135277.log.gz 2023-08-26T18:17:50.571 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.116025.log 2023-08-26T18:17:50.587 INFO:teuthology.orchestra.run.smithi124.stderr: 93.7% -- replaced with /var/log/ceph/cluster2-client.mirror.115570.log.gz 2023-08-26T18:17:50.597 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster2-client.mirror.116025.log: 87.4% -- replaced with /var/log/ceph/cluster2-client.mirror.116025.log.gz 2023-08-26T18:17:50.597 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.117250.log 2023-08-26T18:17:50.597 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster2-client.mirror.117250.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.106772.log 2023-08-26T18:17:50.608 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster2-client.admin.106772.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.106772.log.gz 2023-08-26T18:17:50.608 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.133797.log 2023-08-26T18:17:50.619 INFO:teuthology.orchestra.run.smithi124.stderr: 93.7% -- replaced with /var/log/ceph/cluster2-client.mirror.141412.log.gz 2023-08-26T18:17:50.630 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster2-client.mirror.133797.log: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.133797.log.gz 2023-08-26T18:17:50.630 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.142498.log 2023-08-26T18:17:50.630 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster1-client.mirror.142498.log: 86.8% -- replaced with /var/log/ceph/cluster1-client.mirror.142498.log.gz 2023-08-26T18:17:50.630 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.140663.log 2023-08-26T18:17:50.631 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster2-client.mirror.140663.log: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.140663.log.gz 2023-08-26T18:17:50.631 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.138298.log 2023-08-26T18:17:50.631 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster1-client.mirror.138298.log: 86.8% -- replaced with /var/log/ceph/cluster1-client.mirror.138298.log.gz 2023-08-26T18:17:50.632 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.118127.log 2023-08-26T18:17:50.632 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster2-client.mirror.118127.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.108269.log 2023-08-26T18:17:50.641 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster2-client.mirror.108269.log: 89.7% -- replaced with /var/log/ceph/cluster2-client.mirror.108269.log.gz 2023-08-26T18:17:50.642 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.112066.log 2023-08-26T18:17:50.646 INFO:teuthology.orchestra.run.smithi124.stderr: 93.7% -- replaced with /var/log/ceph/cluster2-client.mirror.115663.log.gz 2023-08-26T18:17:50.654 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster2-client.mirror.112066.log: 93.7% -- replaced with /var/log/ceph/cluster2-client.mirror.139181.log.gz 2023-08-26T18:17:50.654 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.136395.log 2023-08-26T18:17:50.662 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster2-client.mirror.136395.log: 90.5% -- replaced with /var/log/ceph/cluster2-client.mirror.136395.log.gz 2023-08-26T18:17:50.662 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.134491.log 2023-08-26T18:17:50.663 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster2-client.mirror.134491.log: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.134491.log.gz 2023-08-26T18:17:50.663 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.129471.log 2023-08-26T18:17:50.664 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster1-client.mirror.129471.log: 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.129471.log.gz 2023-08-26T18:17:50.664 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.141094.log 2023-08-26T18:17:50.665 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster2-client.mirror.141094.log: 79.4% -- replaced with /var/log/ceph/cluster2-client.mirror.141094.log.gz 2023-08-26T18:17:50.665 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.113921.log 2023-08-26T18:17:50.672 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster1-client.mirror.113921.log: 93.4% -- replaced with /var/log/ceph/cluster1-client.mirror.113921.log.gz 2023-08-26T18:17:50.672 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.138905.log 2023-08-26T18:17:50.673 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster1-client.mirror.138905.log: 86.7% -- replaced with /var/log/ceph/cluster1-client.mirror.138905.log.gz 2023-08-26T18:17:50.673 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.125829.log 2023-08-26T18:17:50.673 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster2-client.mirror.125829.log: 87.4% -- replaced with /var/log/ceph/cluster2-client.mirror.125829.log.gz 2023-08-26T18:17:50.674 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.143542.log 2023-08-26T18:17:50.675 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster2-client.mirror.143542.log: 8.2% -- replaced with /var/log/ceph/cluster2-client.mirror.143542.log.gz 2023-08-26T18:17:50.675 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.136581.log 2023-08-26T18:17:50.682 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster2-client.mirror.136581.log: 90.4% -- replaced with /var/log/ceph/cluster2-client.mirror.136581.log.gz 2023-08-26T18:17:50.682 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.117939.log 2023-08-26T18:17:50.691 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster2-client.mirror.117939.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.124481.log 2023-08-26T18:17:50.710 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster1-client.mirror.124481.log: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.124481.log.gz 2023-08-26T18:17:50.710 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.112322.log 2023-08-26T18:17:50.712 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster1-client.mirror.112322.log: 89.0% -- replaced with /var/log/ceph/cluster1-client.mirror.112322.log.gz 2023-08-26T18:17:50.712 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.126325.log 2023-08-26T18:17:50.728 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster2-client.mirror.126325.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.0.141641.log 2023-08-26T18:17:50.728 INFO:teuthology.orchestra.run.smithi124.stderr: 89.7% -- replaced with /var/log/ceph/cluster2-client.mirror.126325.log.gz 2023-08-26T18:17:50.740 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster1-client.mirror.0.141641.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.134864.log 2023-08-26T18:17:50.754 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster2-client.mirror.134864.log: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.134864.log.gz 2023-08-26T18:17:50.754 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.127650.log 2023-08-26T18:17:50.768 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster1-client.mirror.127650.log: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.127650.log.gz 2023-08-26T18:17:50.768 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.136093.log 2023-08-26T18:17:50.793 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster2-client.mirror.136093.log: 88.0% -- replaced with /var/log/ceph/cluster2-client.mirror.136093.log.gz 2023-08-26T18:17:50.793 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.128059.log 2023-08-26T18:17:50.804 INFO:teuthology.orchestra.run.smithi124.stderr: 95.6% -- replaced with /var/log/ceph/cluster1-client.mirror.0.141641.log.gz 2023-08-26T18:17:50.815 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster1-client.mirror.128059.log: 94.1% -- replaced with /var/log/ceph/cluster1-client.mirror.128059.log.gz 2023-08-26T18:17:50.815 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.109015.log 2023-08-26T18:17:50.823 INFO:teuthology.orchestra.run.smithi124.stderr: 93.7% -- replaced with /var/log/ceph/cluster2-client.mirror.117250.log.gz 2023-08-26T18:17:50.833 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster1-client.mirror.109015.log: 87.5% -- replaced with /var/log/ceph/cluster1-client.mirror.109015.log.gz 2023-08-26T18:17:50.833 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.109858.log 2023-08-26T18:17:50.833 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster2-client.mirror.109858.log: 89.7% -- replaced with /var/log/ceph/cluster2-client.mirror.109858.log.gz 2023-08-26T18:17:50.834 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.135380.log 2023-08-26T18:17:50.834 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster2-client.mirror.135380.log: 90.4% -- replaced with /var/log/ceph/cluster2-client.mirror.135380.log.gz 2023-08-26T18:17:50.834 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.121461.log 2023-08-26T18:17:50.840 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster1-client.mirror.121461.log: 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.121461.log.gz 2023-08-26T18:17:50.840 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.121321.log 2023-08-26T18:17:50.841 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster2-client.mirror.121321.log: 89.9% -- replaced with /var/log/ceph/cluster2-client.mirror.121321.log.gz 2023-08-26T18:17:50.841 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.2.106228.log 2023-08-26T18:17:50.852 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster2-client.mirror.2.106228.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.2.119890.log 2023-08-26T18:17:50.878 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster2-client.mirror.2.119890.log: 92.7% -- replaced with /var/log/ceph/cluster2-client.mirror.2.106228.log.gz 2023-08-26T18:17:50.878 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.141466.log 2023-08-26T18:17:50.879 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster1-client.mirror.141466.log: 87.2% -- replaced with /var/log/ceph/cluster1-client.mirror.141466.log.gz 2023-08-26T18:17:50.879 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.108985.log 2023-08-26T18:17:50.889 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster1-client.admin.108985.log: 87.5% -- replaced with /var/log/ceph/cluster1-client.admin.108985.log.gz 2023-08-26T18:17:50.890 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.134511.log 2023-08-26T18:17:50.909 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster2-client.mirror.134511.log: 90.4% -- replaced with /var/log/ceph/cluster2-client.mirror.134511.log.gz 2023-08-26T18:17:50.910 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.125327.log 2023-08-26T18:17:50.910 INFO:teuthology.orchestra.run.smithi124.stderr: 93.7% -- replaced with /var/log/ceph/cluster2-client.mirror.112066.log.gz 2023-08-26T18:17:50.911 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster2-client.mirror.125327.log: 89.8% -- replaced with /var/log/ceph/cluster2-client.mirror.125327.log.gz 2023-08-26T18:17:50.916 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.137330.log 2023-08-26T18:17:50.916 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster1-client.mirror.137330.log: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.137330.log.gz 2023-08-26T18:17:50.924 INFO:teuthology.orchestra.run.smithi124.stderr: 94.0% -- replaced with /var/log/ceph/cluster2-client.mirror.2.119890.log.gz 2023-08-26T18:17:50.925 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.111914.log 2023-08-26T18:17:50.926 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster1-client.mirror.111914.log: 88.9% -- replaced with /var/log/ceph/cluster1-client.mirror.111914.log.gz 2023-08-26T18:17:50.926 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.133217.log 2023-08-26T18:17:50.927 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster1-client.mirror.133217.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.133217.log.gz 2023-08-26T18:17:50.927 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.126392.log 2023-08-26T18:17:50.928 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster2-client.mirror.126392.log: 87.4% -- replaced with /var/log/ceph/cluster2-client.mirror.126392.log.gz 2023-08-26T18:17:50.928 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.136602.log 2023-08-26T18:17:50.929 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster2-client.mirror.136602.log: 88.0% -- replaced with /var/log/ceph/cluster2-client.mirror.136602.log.gz 2023-08-26T18:17:50.929 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.111570.log 2023-08-26T18:17:50.930 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster1-client.admin.111570.log: 87.8% -- replaced with /var/log/ceph/cluster1-client.admin.111570.log.gz 2023-08-26T18:17:50.931 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.106728.log 2023-08-26T18:17:50.931 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster1-client.mirror.106728.log: 54.5% -- replaced with /var/log/ceph/cluster1-client.mirror.106728.log.gz 2023-08-26T18:17:50.932 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.110198.log 2023-08-26T18:17:50.933 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster1-client.mirror.110198.log: 88.3% -- replaced with /var/log/ceph/cluster1-client.mirror.110198.log.gz 2023-08-26T18:17:50.933 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.140619.log 2023-08-26T18:17:50.933 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster1-client.mirror.140619.log: 86.8% -- replaced with /var/log/ceph/cluster1-client.mirror.140619.log.gz 2023-08-26T18:17:50.934 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.129399.log 2023-08-26T18:17:50.935 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster1-client.mirror.129399.log: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.129399.log.gz 2023-08-26T18:17:50.935 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.121127.log 2023-08-26T18:17:50.935 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster1-client.mirror.121127.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.121127.log.gz 2023-08-26T18:17:50.935 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.119326.log 2023-08-26T18:17:50.938 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster2-client.mirror.119326.log: 93.7% -- replaced with /var/log/ceph/cluster2-client.mirror.118127.log.gz 2023-08-26T18:17:50.948 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.120663.log 2023-08-26T18:17:50.948 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster1-client.mirror.120663.log: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.120663.log.gz 2023-08-26T18:17:50.948 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.108504.log 2023-08-26T18:17:50.949 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster1-client.mirror.108504.log: 87.6% -- replaced with /var/log/ceph/cluster1-client.mirror.108504.log.gz 2023-08-26T18:17:50.949 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.112546.log 2023-08-26T18:17:50.957 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster2-client.mirror.112546.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.129331.log 2023-08-26T18:17:50.959 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster2-client.mirror.129331.log: 89.7% -- replaced with /var/log/ceph/cluster2-client.mirror.129331.log.gz 2023-08-26T18:17:50.959 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.120088.log 2023-08-26T18:17:50.960 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster2-client.mirror.120088.log: 79.5% -- replaced with /var/log/ceph/cluster2-client.mirror.120088.log.gz 2023-08-26T18:17:50.960 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.128873.log 2023-08-26T18:17:50.961 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster2-client.mirror.128873.log: 89.7% -- replaced with /var/log/ceph/cluster2-client.mirror.128873.log.gz 2023-08-26T18:17:50.961 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.123180.log 2023-08-26T18:17:50.963 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster2-client.mirror.123180.log: 89.7% -- replaced with /var/log/ceph/cluster2-client.mirror.123180.log.gz 2023-08-26T18:17:50.963 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.141840.log 2023-08-26T18:17:50.964 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster2-client.mirror.141840.log: 79.4% -- replaced with /var/log/ceph/cluster2-client.mirror.141840.log.gz 2023-08-26T18:17:50.964 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.123715.log 2023-08-26T18:17:50.965 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster2-client.mirror.123715.log: 94.0% -- replaced with /var/log/ceph/cluster2-client.mirror.112546.log.gz 2023-08-26T18:17:50.966 INFO:teuthology.orchestra.run.smithi124.stderr: 89.8% -- replaced with /var/log/ceph/cluster2-client.mirror.123715.log.gz 2023-08-26T18:17:50.966 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.130113.log 2023-08-26T18:17:50.967 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster2-client.mirror.130113.log: 89.7% -- replaced with /var/log/ceph/cluster2-client.mirror.130113.log.gz 2023-08-26T18:17:50.967 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.120639.log 2023-08-26T18:17:50.968 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster1-client.mirror.120639.log: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.120639.log.gz 2023-08-26T18:17:50.968 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.142857.log 2023-08-26T18:17:50.970 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster1-client.mirror.142857.log: 67.9% -- replaced with /var/log/ceph/cluster1-client.mirror.142857.log.gz 2023-08-26T18:17:50.971 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.106694.log 2023-08-26T18:17:50.971 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster1-client.mirror.106694.log: 56.7% -- replaced with /var/log/ceph/cluster1-client.mirror.106694.log.gz 2023-08-26T18:17:50.971 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.130091.log 2023-08-26T18:17:50.972 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster1-client.mirror.130091.log: 91.9% -- replaced with /var/log/ceph/cluster1-client.mirror.130091.log.gz 2023-08-26T18:17:50.972 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.109656.log 2023-08-26T18:17:50.973 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster1-client.mirror.109656.log: 88.0% -- replaced with /var/log/ceph/cluster1-client.mirror.109656.log.gz 2023-08-26T18:17:50.974 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.138095.log 2023-08-26T18:17:50.974 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster1-client.mirror.138095.log: 86.8% -- replaced with /var/log/ceph/cluster1-client.mirror.138095.log.gz 2023-08-26T18:17:50.974 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.141877.log 2023-08-26T18:17:50.976 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster2-client.mirror.141877.log: 87.4% -- replaced with /var/log/ceph/cluster2-client.mirror.141877.log.gz 2023-08-26T18:17:50.976 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.119415.log 2023-08-26T18:17:50.982 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster2-client.mirror.119415.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.119168.log 2023-08-26T18:17:50.993 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster2-client.mirror.119168.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.140424.log 2023-08-26T18:17:51.010 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster2-client.mirror.140424.log: 88.1% -- replaced with /var/log/ceph/cluster2-client.mirror.140424.log.gz 2023-08-26T18:17:51.010 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.118921.log 2023-08-26T18:17:51.032 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster2-client.mirror.118921.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.115226.log 2023-08-26T18:17:51.043 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster2-client.mirror.115226.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.126862.log 2023-08-26T18:17:51.043 INFO:teuthology.orchestra.run.smithi124.stderr: 89.6% -- replaced with /var/log/ceph/cluster2-client.mirror.115226.log.gz 2023-08-26T18:17:51.062 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster2-client.mirror.126862.log: 87.3% -- replaced with /var/log/ceph/cluster2-client.mirror.126862.log.gz 2023-08-26T18:17:51.062 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.140317.log 2023-08-26T18:17:51.082 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster1-client.mirror.140317.log: 87.1% -- replaced with /var/log/ceph/cluster1-client.mirror.140317.log.gz 2023-08-26T18:17:51.083 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.141004.log 2023-08-26T18:17:51.095 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster1-client.mirror.141004.log: 76.3% -- replaced with /var/log/ceph/cluster1-client.mirror.141004.log.gz 2023-08-26T18:17:51.095 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.132614.log 2023-08-26T18:17:51.104 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster2-client.mirror.132614.log: 89.8% -- replaced with /var/log/ceph/cluster2-client.mirror.132614.log.gz 2023-08-26T18:17:51.104 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.119800.log 2023-08-26T18:17:51.116 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster2-client.mirror.119800.log: 89.7% -- replaced with /var/log/ceph/cluster2-client.mirror.119800.log.gz 2023-08-26T18:17:51.118 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.111346.log 2023-08-26T18:17:51.118 INFO:teuthology.orchestra.run.smithi124.stderr: 93.7% -- replaced with /var/log/ceph/cluster2-client.mirror.117939.log.gz 2023-08-26T18:17:51.119 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster1-client.mirror.111346.log: 88.7% -- replaced with /var/log/ceph/cluster1-client.mirror.111346.log.gz 2023-08-26T18:17:51.120 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.133282.log 2023-08-26T18:17:51.122 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster1-client.mirror.133282.log: 92.3% -- replaced with /var/log/ceph/cluster1-client.mirror.133282.log.gz 2023-08-26T18:17:51.122 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.138703.log 2023-08-26T18:17:51.123 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster1-client.mirror.138703.log: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.138703.log.gz 2023-08-26T18:17:51.123 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.128917.log 2023-08-26T18:17:51.124 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster1-client.mirror.128917.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.128917.log.gz 2023-08-26T18:17:51.124 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.105053.log 2023-08-26T18:17:51.125 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster2-client.admin.105053.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.105053.log.gz 2023-08-26T18:17:51.125 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.105836.log 2023-08-26T18:17:51.144 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster2-client.admin.105836.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.105836.log.gz 2023-08-26T18:17:51.144 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.142669.log 2023-08-26T18:17:51.156 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster1-client.admin.142669.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.142669.log.gz 2023-08-26T18:17:51.157 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.122668.log 2023-08-26T18:17:51.181 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster2-client.mirror.122668.log: 92.4% -- replaced with /var/log/ceph/cluster2-client.mirror.122668.log.gz 2023-08-26T18:17:51.181 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.123831.log 2023-08-26T18:17:51.192 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster1-client.mirror.123831.log: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.123831.log.gz 2023-08-26T18:17:51.192 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.128199.log 2023-08-26T18:17:51.203 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster1-client.mirror.128199.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.128199.log.gz 2023-08-26T18:17:51.203 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.140577.log 2023-08-26T18:17:51.215 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster1-client.mirror.140577.log: 87.2% -- replaced with /var/log/ceph/cluster1-client.mirror.140577.log.gz 2023-08-26T18:17:51.216 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.111394.log 2023-08-26T18:17:51.229 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster1-client.mirror.111394.log: 88.6% -- replaced with /var/log/ceph/cluster1-client.mirror.111394.log.gz 2023-08-26T18:17:51.229 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.120125.log 2023-08-26T18:17:51.241 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster2-client.mirror.120125.log: 87.5% -- replaced with /var/log/ceph/cluster2-client.mirror.120125.log.gz 2023-08-26T18:17:51.241 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.129744.log 2023-08-26T18:17:51.266 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster1-client.mirror.129744.log: 92.4% -- replaced with /var/log/ceph/cluster1-client.mirror.129744.log.gz 2023-08-26T18:17:51.266 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.116901.log 2023-08-26T18:17:51.277 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster2-client.mirror.116901.log: 87.4% -- replaced with /var/log/ceph/cluster2-client.mirror.116901.log.gz 2023-08-26T18:17:51.277 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.116752.log 2023-08-26T18:17:51.288 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster2-client.mirror.116752.log: 79.7% -- replaced with /var/log/ceph/cluster2-client.mirror.116752.log.gz 2023-08-26T18:17:51.288 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.133672.log 2023-08-26T18:17:51.302 INFO:teuthology.orchestra.run.smithi124.stderr: 93.7% -- replaced with /var/log/ceph/cluster2-client.mirror.119415.log.gz 2023-08-26T18:17:51.302 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster2-client.mirror.133672.log: 88.1% -- replaced with /var/log/ceph/cluster2-client.mirror.133672.log.gz 2023-08-26T18:17:51.314 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.127364.log 2023-08-26T18:17:51.314 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster1-client.mirror.127364.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.127364.log.gz 2023-08-26T18:17:51.314 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.106228.log 2023-08-26T18:17:51.315 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster1-client.mirror.106228.log: 93.1% -- replaced with /var/log/ceph/cluster1-client.mirror.106228.log.gz 2023-08-26T18:17:51.315 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.109880.log 2023-08-26T18:17:51.323 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster2-client.mirror.109880.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.142179.log 2023-08-26T18:17:51.324 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster1-client.mirror.142179.log: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.142179.log.gz 2023-08-26T18:17:51.324 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.114841.log 2023-08-26T18:17:51.343 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.1.116971.log 2023-08-26T18:17:51.351 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster1-client.mirror.1.116971.log: /var/log/ceph/cluster1-client.mirror.114841.log: 86.6% -- replaced with /var/log/ceph/cluster1-client.mirror.114841.log.gz 2023-08-26T18:17:51.354 INFO:teuthology.orchestra.run.smithi124.stderr: 93.7% -- replaced with /var/log/ceph/cluster2-client.mirror.118921.log.gz 2023-08-26T18:17:51.363 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.138613.log 2023-08-26T18:17:51.363 INFO:teuthology.orchestra.run.smithi124.stderr: 93.7% -- replaced with /var/log/ceph/cluster2-client.mirror.119326.log.gz 2023-08-26T18:17:51.364 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster1-client.mirror.138613.log: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.138613.log.gz 2023-08-26T18:17:51.364 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.109210.log 2023-08-26T18:17:51.366 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster1-client.mirror.109210.log: 87.8% -- replaced with /var/log/ceph/cluster1-client.mirror.109210.log.gz 2023-08-26T18:17:51.367 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.0.106005.log 2023-08-26T18:17:51.367 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster1-client.mirror.0.106005.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.116258.log 2023-08-26T18:17:51.385 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster2-client.mirror.116258.log: 89.6% -- replaced with /var/log/ceph/cluster2-client.mirror.116258.log.gz 2023-08-26T18:17:51.385 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.108899.log 2023-08-26T18:17:51.387 INFO:teuthology.orchestra.run.smithi124.stderr: 93.7% -- replaced with /var/log/ceph/cluster2-client.mirror.119168.log.gz 2023-08-26T18:17:51.389 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster1-client.admin.108899.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.admin.108899.log.gz 2023-08-26T18:17:51.390 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.119225.log 2023-08-26T18:17:51.390 INFO:teuthology.orchestra.run.smithi124.stderr: 94.1% -- replaced with /var/log/ceph/cluster1-client.mirror.0.106005.log.gz 2023-08-26T18:17:51.391 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster2-client.mirror.119225.log: 87.5% -- replaced with /var/log/ceph/cluster2-client.mirror.119225.log.gz 2023-08-26T18:17:51.391 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.128312.log 2023-08-26T18:17:51.393 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster1-client.mirror.128312.log: 92.2% -- replaced with /var/log/ceph/cluster1-client.mirror.128312.log.gz 2023-08-26T18:17:51.394 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.108951.log 2023-08-26T18:17:51.394 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster1-client.admin.108951.log: 87.5% -- replaced with /var/log/ceph/cluster1-client.admin.108951.log.gz 2023-08-26T18:17:51.394 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.131422.log 2023-08-26T18:17:51.395 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster1-client.mirror.131422.log: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.131422.log.gz 2023-08-26T18:17:51.395 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.131695.log 2023-08-26T18:17:51.396 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.118526.log 2023-08-26T18:17:51.396 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster2-client.mirror.131695.log: /var/log/ceph/cluster2-client.mirror.118526.log: 89.8% -- replaced with /var/log/ceph/cluster2-client.mirror.131695.log.gz 2023-08-26T18:17:51.397 INFO:teuthology.orchestra.run.smithi124.stderr: 89.7% -- replaced with /var/log/ceph/cluster2-client.mirror.118526.log.gz 2023-08-26T18:17:51.397 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.135008.log 2023-08-26T18:17:51.398 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster2-client.mirror.135008.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.104916.log 2023-08-26T18:17:51.398 INFO:teuthology.orchestra.run.smithi124.stderr: 90.4% -- replaced with /var/log/ceph/cluster2-client.mirror.135008.log.gz 2023-08-26T18:17:51.399 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster2-client.admin.104916.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.104916.log.gz 2023-08-26T18:17:51.400 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.142950.log 2023-08-26T18:17:51.400 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.136292.log 2023-08-26T18:17:51.401 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster1-client.admin.142950.log: /var/log/ceph/cluster2-client.mirror.136292.log: 70.5% -- replaced with /var/log/ceph/cluster1-client.admin.142950.log.gz 2023-08-26T18:17:51.401 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.121591.log 2023-08-26T18:17:51.401 INFO:teuthology.orchestra.run.smithi124.stderr: 88.0% -- replaced with /var/log/ceph/cluster2-client.mirror.136292.log.gz 2023-08-26T18:17:51.405 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster2-client.mirror.121591.log: 87.3% -- replaced with /var/log/ceph/cluster2-client.mirror.121591.log.gz 2023-08-26T18:17:51.406 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.132503.log 2023-08-26T18:17:51.406 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster1-client.mirror.132503.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.132503.log.gz 2023-08-26T18:17:51.407 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.141370.log 2023-08-26T18:17:51.408 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster2-client.mirror.141370.log: 89.6% -- replaced with /var/log/ceph/cluster2-client.mirror.141370.log.gz 2023-08-26T18:17:51.408 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.143750.log 2023-08-26T18:17:51.413 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.136970.log 2023-08-26T18:17:51.414 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster2-client.mirror.143750.log: 6.1% -- replaced with /var/log/ceph/cluster2-client.mirror.143750.log.gz 2023-08-26T18:17:51.415 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster1-client.mirror.136970.log: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.136970.log.gz 2023-08-26T18:17:51.415 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.120337.log 2023-08-26T18:17:51.416 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.112715.log 2023-08-26T18:17:51.416 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster2-client.mirror.120337.log: /var/log/ceph/cluster1-client.admin.112715.log: 89.8% -- replaced with /var/log/ceph/cluster2-client.mirror.120337.log.gz 2023-08-26T18:17:51.416 INFO:teuthology.orchestra.run.smithi124.stderr: 88.3% -- replaced with /var/log/ceph/cluster1-client.admin.112715.log.gz 2023-08-26T18:17:51.416 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.112044.log 2023-08-26T18:17:51.417 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster2-client.mirror.112044.log: 89.9% -- replaced with /var/log/ceph/cluster2-client.mirror.112044.log.gz 2023-08-26T18:17:51.417 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.142386.log 2023-08-26T18:17:51.420 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.127674.log 2023-08-26T18:17:51.421 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster1-client.mirror.142386.log: 86.8% -- replaced with /var/log/ceph/cluster1-client.mirror.142386.log.gz 2023-08-26T18:17:51.422 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster1-client.mirror.127674.log: 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.127674.log.gz 2023-08-26T18:17:51.422 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.126886.log 2023-08-26T18:17:51.422 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster1-client.mirror.126886.log: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.126886.log.gz 2023-08-26T18:17:51.423 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.127227.log 2023-08-26T18:17:51.424 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster1-client.mirror.127227.log: 92.3% -- replaced with /var/log/ceph/cluster1-client.mirror.127227.log.gz 2023-08-26T18:17:51.424 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.107011.log 2023-08-26T18:17:51.425 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster2-client.mirror.107011.log: 87.4% -- replaced with /var/log/ceph/cluster2-client.mirror.107011.log.gz 2023-08-26T18:17:51.425 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.136793.log 2023-08-26T18:17:51.426 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster1-client.mirror.136793.log: 1.3% -- replaced with /var/log/ceph/cluster1-client.mirror.136793.log.gz 2023-08-26T18:17:51.427 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.127796.log 2023-08-26T18:17:51.427 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster2-client.mirror.127796.log: 89.8% -- replaced with /var/log/ceph/cluster2-client.mirror.127796.log.gz 2023-08-26T18:17:51.427 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.138253.log 2023-08-26T18:17:51.428 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster1-client.mirror.138253.log: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.138253.log.gz 2023-08-26T18:17:51.429 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.126529.log 2023-08-26T18:17:51.430 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster1-client.mirror.126529.log: 92.2% -- replaced with /var/log/ceph/cluster1-client.mirror.126529.log.gz 2023-08-26T18:17:51.430 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.105652.log 2023-08-26T18:17:51.432 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster1-client.admin.105652.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.105652.log.gz 2023-08-26T18:17:51.432 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.141390.log 2023-08-26T18:17:51.433 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster2-client.mirror.141390.log: 87.4% -- replaced with /var/log/ceph/cluster2-client.mirror.141390.log.gz 2023-08-26T18:17:51.433 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.142728.log 2023-08-26T18:17:51.435 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster1-client.admin.142728.log: 70.3% -- replaced with /var/log/ceph/cluster1-client.admin.142728.log.gz 2023-08-26T18:17:51.435 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.138523.log 2023-08-26T18:17:51.435 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster1-client.mirror.138523.log: 86.7% -- replaced with /var/log/ceph/cluster1-client.mirror.138523.log.gz 2023-08-26T18:17:51.436 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.117762.log 2023-08-26T18:17:51.445 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster2-client.mirror.117762.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.125034.log 2023-08-26T18:17:51.451 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster2-client.mirror.125034.log: 89.7% -- replaced with /var/log/ceph/cluster2-client.mirror.125034.log.gz 2023-08-26T18:17:51.453 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.145379.log 2023-08-26T18:17:51.464 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster2-client.admin.145379.log: 7.1% -- replaced with /var/log/ceph/cluster2-client.admin.145379.log.gz 2023-08-26T18:17:51.465 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.115923.log 2023-08-26T18:17:51.466 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster2-client.mirror.115923.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.125559.log 2023-08-26T18:17:51.467 INFO:teuthology.orchestra.run.smithi124.stderr: 89.6% -- replaced with /var/log/ceph/cluster2-client.mirror.115923.log.gz 2023-08-26T18:17:51.468 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster2-client.mirror.125559.log: 89.9% -- replaced with /var/log/ceph/cluster2-client.mirror.125559.log.gz 2023-08-26T18:17:51.468 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.122047.log 2023-08-26T18:17:51.470 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster2-client.mirror.122047.log: 89.8% -- replaced with /var/log/ceph/cluster2-client.mirror.122047.log.gz 2023-08-26T18:17:51.470 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.122575.log 2023-08-26T18:17:51.471 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster1-client.mirror.122575.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.122575.log.gz 2023-08-26T18:17:51.472 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.142521.log 2023-08-26T18:17:51.474 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster1-client.mirror.142521.log: 87.0% -- replaced with /var/log/ceph/cluster1-client.mirror.142521.log.gz 2023-08-26T18:17:51.474 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.131470.log 2023-08-26T18:17:51.474 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster1-client.mirror.131470.log: 87.6% -- replaced with /var/log/ceph/cluster1-client.mirror.131470.log.gz 2023-08-26T18:17:51.474 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.121871.log 2023-08-26T18:17:51.475 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster1-client.mirror.121871.log: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.121871.log.gz 2023-08-26T18:17:51.475 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.126842.log 2023-08-26T18:17:51.476 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster2-client.mirror.126842.log: 89.8% -- replaced with /var/log/ceph/cluster2-client.mirror.126842.log.gz 2023-08-26T18:17:51.476 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.110804.log 2023-08-26T18:17:51.477 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster1-client.mirror.110804.log: 88.4% -- replaced with /var/log/ceph/cluster1-client.mirror.110804.log.gz 2023-08-26T18:17:51.477 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.136993.log 2023-08-26T18:17:51.478 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster1-client.mirror.136993.log: 86.8% -- replaced with /var/log/ceph/cluster1-client.mirror.136993.log.gz 2023-08-26T18:17:51.478 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.135672.log 2023-08-26T18:17:51.479 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster2-client.mirror.135672.log: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.135672.log.gz 2023-08-26T18:17:51.481 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.130071.log 2023-08-26T18:17:51.482 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster2-client.mirror.130071.log: 92.1% -- replaced with /var/log/ceph/cluster2-client.mirror.130071.log.gz 2023-08-26T18:17:51.482 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.123662.log 2023-08-26T18:17:51.483 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster2-client.mirror.123662.log: 89.7% -- replaced with /var/log/ceph/cluster2-client.mirror.123662.log.gz 2023-08-26T18:17:51.484 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.134429.log 2023-08-26T18:17:51.485 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster2-client.mirror.134429.log: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.134429.log.gz 2023-08-26T18:17:51.485 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.112399.log 2023-08-26T18:17:51.490 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster2-client.mirror.112399.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.116532.log 2023-08-26T18:17:51.491 INFO:teuthology.orchestra.run.smithi124.stderr: 93.4% -- replaced with /var/log/ceph/cluster2-client.mirror.112399.log.gz 2023-08-26T18:17:51.493 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster2-client.mirror.116532.log: 89.6% -- replaced with /var/log/ceph/cluster2-client.mirror.116532.log.gz 2023-08-26T18:17:51.493 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.135796.log 2023-08-26T18:17:51.494 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster2-client.mirror.135796.log: 88.1% -- replaced with /var/log/ceph/cluster2-client.mirror.135796.log.gz 2023-08-26T18:17:51.495 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.135194.log 2023-08-26T18:17:51.497 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster2-client.mirror.135194.log: 90.5% -- replaced with /var/log/ceph/cluster2-client.mirror.135194.log.gz 2023-08-26T18:17:51.497 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.138208.log 2023-08-26T18:17:51.498 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster1-client.mirror.138208.log: 86.8% -- replaced with /var/log/ceph/cluster1-client.mirror.138208.log.gz 2023-08-26T18:17:51.498 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.140403.log 2023-08-26T18:17:51.499 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster2-client.mirror.140403.log: 88.1% -- replaced with /var/log/ceph/cluster2-client.mirror.140403.log.gz 2023-08-26T18:17:51.499 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.113878.log 2023-08-26T18:17:51.502 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster1-client.mirror.113878.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.105606.log 2023-08-26T18:17:51.506 INFO:teuthology.orchestra.run.smithi124.stderr: 93.3% -- replaced with /var/log/ceph/cluster1-client.mirror.113878.log.gz 2023-08-26T18:17:51.506 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster1-client.admin.105606.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.105606.log.gz 2023-08-26T18:17:51.507 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.137015.log 2023-08-26T18:17:51.507 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster1-client.mirror.137015.log: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.137015.log.gz 2023-08-26T18:17:51.508 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.132193.log 2023-08-26T18:17:51.508 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster2-client.mirror.132193.log: 87.4% -- replaced with /var/log/ceph/cluster2-client.mirror.132193.log.gz 2023-08-26T18:17:51.508 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.136437.log 2023-08-26T18:17:51.509 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster2-client.mirror.136437.log: 88.0% -- replaced with /var/log/ceph/cluster2-client.mirror.136437.log.gz 2023-08-26T18:17:51.509 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.121341.log 2023-08-26T18:17:51.510 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster2-client.mirror.121341.log: 87.4% -- replaced with /var/log/ceph/cluster2-client.mirror.121341.log.gz 2023-08-26T18:17:51.510 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.136499.log 2023-08-26T18:17:51.511 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster2-client.mirror.136499.log: 88.1% -- replaced with /var/log/ceph/cluster2-client.mirror.136499.log.gz 2023-08-26T18:17:51.512 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.117097.log 2023-08-26T18:17:51.513 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster2-client.mirror.117097.log: 87.3% -- replaced with /var/log/ceph/cluster2-client.mirror.117097.log.gz 2023-08-26T18:17:51.513 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.131852.log 2023-08-26T18:17:51.515 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster1-client.mirror.131852.log: 92.3% -- replaced with /var/log/ceph/cluster1-client.mirror.131852.log.gz 2023-08-26T18:17:51.516 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.134843.log 2023-08-26T18:17:51.516 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster2-client.mirror.134843.log: 87.8% -- replaced with /var/log/ceph/cluster2-client.mirror.134843.log.gz 2023-08-26T18:17:51.517 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.140338.log 2023-08-26T18:17:51.518 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster2-client.mirror.140338.log: 88.6% -- replaced with /var/log/ceph/cluster2-client.mirror.140338.log.gz 2023-08-26T18:17:51.518 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.124923.log 2023-08-26T18:17:51.519 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster1-client.mirror.124923.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.124923.log.gz 2023-08-26T18:17:51.519 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.121199.log 2023-08-26T18:17:51.521 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster1-client.mirror.121199.log: 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.121199.log.gz 2023-08-26T18:17:51.521 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.113706.log 2023-08-26T18:17:51.528 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster1-client.mirror.113706.log: 93.4% -- replaced with /var/log/ceph/cluster1-client.mirror.113706.log.gz 2023-08-26T18:17:51.528 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.125081.log 2023-08-26T18:17:51.529 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster2-client.mirror.125081.log: 89.7% -- replaced with /var/log/ceph/cluster2-client.mirror.125081.log.gz 2023-08-26T18:17:51.529 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.115943.log 2023-08-26T18:17:51.530 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster2-client.mirror.115943.log: 87.4% -- replaced with /var/log/ceph/cluster2-client.mirror.115943.log.gz 2023-08-26T18:17:51.531 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.110150.log 2023-08-26T18:17:51.532 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster2-client.mirror.110150.log: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.110150.log.gz 2023-08-26T18:17:51.532 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.134740.log 2023-08-26T18:17:51.536 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster2-client.mirror.134740.log: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.134740.log.gz 2023-08-26T18:17:51.536 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.126910.log 2023-08-26T18:17:51.536 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster1-client.mirror.126910.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.126910.log.gz 2023-08-26T18:17:51.537 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.131511.log 2023-08-26T18:17:51.537 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster1-client.mirror.131511.log: 92.4% -- replaced with /var/log/ceph/cluster1-client.mirror.131511.log.gz 2023-08-26T18:17:51.538 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.129954.log 2023-08-26T18:17:51.538 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster2-client.mirror.129954.log: 87.4% -- replaced with /var/log/ceph/cluster2-client.mirror.129954.log.gz 2023-08-26T18:17:51.538 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.108738.log 2023-08-26T18:17:51.539 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster1-client.admin.108738.log: 87.3% -- replaced with /var/log/ceph/cluster1-client.admin.108738.log.gz 2023-08-26T18:17:51.539 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.128792.log 2023-08-26T18:17:51.540 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster1-client.mirror.128792.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.119625.log 2023-08-26T18:17:51.541 INFO:teuthology.orchestra.run.smithi124.stderr: 92.3% -- replaced with /var/log/ceph/cluster1-client.mirror.128792.log.gz 2023-08-26T18:17:51.543 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster2-client.mirror.119625.log: 79.4% -- replaced with /var/log/ceph/cluster2-client.mirror.119625.log.gz 2023-08-26T18:17:51.543 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.111298.log 2023-08-26T18:17:51.543 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster2-client.mirror.111298.log: 88.7% -- replaced with /var/log/ceph/cluster2-client.mirror.111298.log.gz 2023-08-26T18:17:51.544 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.122091.log 2023-08-26T18:17:51.544 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster1-client.mirror.122091.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.122091.log.gz 2023-08-26T18:17:51.545 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.116389.log 2023-08-26T18:17:51.547 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster2-client.mirror.116389.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.127626.log 2023-08-26T18:17:51.558 INFO:teuthology.orchestra.run.smithi124.stderr: 90.1% -- replaced with /var/log/ceph/cluster2-mon.a.log.gz 2023-08-26T18:17:51.560 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster1-client.mirror.127626.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.127626.log.gz 2023-08-26T18:17:51.566 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.132241.log 2023-08-26T18:17:51.568 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster1-client.mirror.132241.log: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.132241.log.gz 2023-08-26T18:17:51.568 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.137218.log 2023-08-26T18:17:51.570 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster1-client.mirror.137218.log: 86.8% -- replaced with /var/log/ceph/cluster1-client.mirror.137218.log.gz 2023-08-26T18:17:51.570 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.125133.log 2023-08-26T18:17:51.571 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster1-client.mirror.125133.log: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.125133.log.gz 2023-08-26T18:17:51.571 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.127912.log 2023-08-26T18:17:51.572 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster1-client.mirror.127912.log: 87.6% -- replaced with /var/log/ceph/cluster1-client.mirror.127912.log.gz 2023-08-26T18:17:51.573 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.116453.log 2023-08-26T18:17:51.574 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster2-client.mirror.116453.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.123001.log 2023-08-26T18:17:51.574 INFO:teuthology.orchestra.run.smithi124.stderr: 89.7% -- replaced with /var/log/ceph/cluster2-client.mirror.116453.log.gz 2023-08-26T18:17:51.576 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster2-client.mirror.123001.log: 89.9% -- replaced with /var/log/ceph/cluster2-client.mirror.123001.log.gz 2023-08-26T18:17:51.576 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.143207.log 2023-08-26T18:17:51.577 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster1-client.mirror.143207.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.143207.log.gz 2023-08-26T18:17:51.578 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.106592.log 2023-08-26T18:17:51.578 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster1-client.admin.106592.log: 79.8% -- replaced with /var/log/ceph/cluster1-client.admin.106592.log.gz 2023-08-26T18:17:51.579 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.117880.log 2023-08-26T18:17:51.579 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster2-client.mirror.117880.log: 79.6% -- replaced with /var/log/ceph/cluster2-client.mirror.117880.log.gz 2023-08-26T18:17:51.580 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.135091.log 2023-08-26T18:17:51.580 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster2-client.mirror.135091.log: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.135091.log.gz 2023-08-26T18:17:51.580 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.134177.log 2023-08-26T18:17:51.581 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster2-client.mirror.134177.log: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.134177.log.gz 2023-08-26T18:17:51.582 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.144023.log 2023-08-26T18:17:51.582 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster2-client.mirror.144023.log: 8.2% -- replaced with /var/log/ceph/cluster2-client.mirror.144023.log.gz 2023-08-26T18:17:51.583 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.113557.log 2023-08-26T18:17:51.584 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster2-client.mirror.113557.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.126666.log 2023-08-26T18:17:51.588 INFO:teuthology.orchestra.run.smithi124.stderr: 93.4% -- replaced with /var/log/ceph/cluster2-client.mirror.113557.log.gz 2023-08-26T18:17:51.590 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster1-client.mirror.126666.log: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.126666.log.gz 2023-08-26T18:17:51.590 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.118862.log 2023-08-26T18:17:51.590 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster2-client.mirror.118862.log: 79.6% -- replaced with /var/log/ceph/cluster2-client.mirror.118862.log.gz 2023-08-26T18:17:51.591 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.137668.log 2023-08-26T18:17:51.592 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster1-client.mirror.137668.log: 86.8% -- replaced with /var/log/ceph/cluster1-client.mirror.137668.log.gz 2023-08-26T18:17:51.593 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.137758.log 2023-08-26T18:17:51.593 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster1-client.mirror.137758.log: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.137758.log.gz 2023-08-26T18:17:51.593 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.114397.log 2023-08-26T18:17:51.595 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster2-client.mirror.114397.log: 88.6% -- replaced with /var/log/ceph/cluster2-client.mirror.114397.log.gz 2023-08-26T18:17:51.595 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.121482.log 2023-08-26T18:17:51.598 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster2-client.mirror.121482.log: 92.4% -- replaced with /var/log/ceph/cluster2-client.mirror.121482.log.gz 2023-08-26T18:17:51.598 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.141153.log 2023-08-26T18:17:51.609 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster2-client.mirror.141153.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.106005.log 2023-08-26T18:17:51.633 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster2-client.mirror.106005.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.121803.log 2023-08-26T18:17:51.644 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster2-client.mirror.121803.log: 89.8% -- replaced with /var/log/ceph/cluster2-client.mirror.121803.log.gz 2023-08-26T18:17:51.645 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.125699.log 2023-08-26T18:17:51.647 INFO:teuthology.orchestra.run.smithi124.stderr: 92.8% -- replaced with /var/log/ceph/cluster1-client.mirror.1.116971.log.gz 2023-08-26T18:17:51.648 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster1-client.mirror.125699.log: 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.125699.log.gz 2023-08-26T18:17:51.652 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.134594.log 2023-08-26T18:17:51.667 INFO:teuthology.orchestra.run.smithi124.stderr: 93.7% -- replaced with /var/log/ceph/cluster2-client.mirror.117762.log.gz 2023-08-26T18:17:51.668 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster2-client.mirror.134594.log: gzip 87.4%-5 -- replaced with /var/log/ceph/cluster2-client.mirror.134594.log.gz--verbose 2023-08-26T18:17:51.668 INFO:teuthology.orchestra.run.smithi124.stderr:-- /var/log/ceph/cluster2-client.mirror.2.141750.log 2023-08-26T18:17:51.668 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster2-client.mirror.2.141750.log: 94.0% -- replaced with /var/log/ceph/cluster2-client.mirror.106005.log.gz 2023-08-26T18:17:51.674 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.127062.log 2023-08-26T18:17:51.675 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster2-client.mirror.127062.log: 89.9% -- replaced with /var/log/ceph/cluster2-client.mirror.127062.log.gz 2023-08-26T18:17:51.676 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.126999.log 2023-08-26T18:17:51.687 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster1-client.mirror.126999.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.114285.log 2023-08-26T18:17:51.688 INFO:teuthology.orchestra.run.smithi124.stderr: 92.3% -- replaced with /var/log/ceph/cluster1-client.mirror.126999.log.gz 2023-08-26T18:17:51.700 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster1-client.mirror.114285.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.0.129807.log 2023-08-26T18:17:51.704 INFO:teuthology.orchestra.run.smithi124.stderr: 95.6% -- replaced with /var/log/ceph/cluster2-client.mirror.2.141750.log.gz 2023-08-26T18:17:51.705 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster2-client.mirror.0.129807.log: 93.4% -- replaced with /var/log/ceph/cluster1-client.mirror.114285.log.gz 2023-08-26T18:17:51.709 INFO:teuthology.orchestra.run.smithi124.stderr: 93.9% -- replaced with /var/log/ceph/cluster2-client.mirror.0.129807.log.gz 2023-08-26T18:17:51.710 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.119763.log 2023-08-26T18:17:51.711 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.126303.log 2023-08-26T18:17:51.711 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.138163.log 2023-08-26T18:17:51.712 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster1-client.mirror.126303.log: /var/log/ceph/cluster1-client.mirror.138163.log: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.138163.log.gz 2023-08-26T18:17:51.712 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.135506.log 2023-08-26T18:17:51.713 INFO:teuthology.orchestra.run.smithi124.stderr: 92.2%/var/log/ceph/cluster2-client.mirror.135506.log: -- replaced with /var/log/ceph/cluster1-client.mirror.126303.log.gz 2023-08-26T18:17:51.723 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster2-client.mirror.119763.log: 90.4% -- replaced with /var/log/ceph/cluster2-client.mirror.135506.log.gz 2023-08-26T18:17:51.723 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.122163.log 2023-08-26T18:17:51.723 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster1-client.mirror.122163.log: 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.122163.log.gz 2023-08-26T18:17:51.724 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.106952.log 2023-08-26T18:17:51.724 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster2-client.mirror.106952.log: 54.2% -- replaced with /var/log/ceph/cluster2-client.mirror.106952.log.gz 2023-08-26T18:17:51.724 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.136230.log 2023-08-26T18:17:51.724 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster2-client.mirror.136230.log: 87.9% -- replaced with /var/log/ceph/cluster2-client.mirror.136230.log.gz 2023-08-26T18:17:51.725 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.125395.log 2023-08-26T18:17:51.725 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster1-client.mirror.125395.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.125395.log.gz 2023-08-26T18:17:51.725 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.122379.log 2023-08-26T18:17:51.726 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster1-client.mirror.122379.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.122379.log.gz 2023-08-26T18:17:51.726 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.106251.log 2023-08-26T18:17:51.726 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster1-client.mirror.106251.log: 93.8% -- replaced with /var/log/ceph/cluster1-client.mirror.106251.log.gz 2023-08-26T18:17:51.726 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.104778.log 2023-08-26T18:17:51.727 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster2-client.admin.104778.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.104778.log.gz 2023-08-26T18:17:51.727 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.115396.log 2023-08-26T18:17:51.736 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster2-client.mirror.115396.log: 79.6% -- replaced with /var/log/ceph/cluster2-client.mirror.115396.log.gz 2023-08-26T18:17:51.736 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.134305.log 2023-08-26T18:17:51.736 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster2-client.mirror.134305.log: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.134305.log.gz 2023-08-26T18:17:51.737 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.144747.log 2023-08-26T18:17:51.737 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster2-client.admin.144747.log: 7.1% -- replaced with /var/log/ceph/cluster2-client.admin.144747.log.gz 2023-08-26T18:17:51.737 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.131110.log 2023-08-26T18:17:51.747 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster2-client.mirror.131110.log: 89.9% -- replaced with /var/log/ceph/cluster2-client.mirror.131110.log.gz 2023-08-26T18:17:51.747 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.115334.log 2023-08-26T18:17:51.747 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster2-client.mirror.115334.log: 87.5% -- replaced with /var/log/ceph/cluster2-client.mirror.115334.log.gz 2023-08-26T18:17:51.748 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.133605.log 2023-08-26T18:17:51.748 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster2-client.mirror.133605.log: 92.5% -- replaced with /var/log/ceph/cluster2-client.mirror.133605.log.gz 2023-08-26T18:17:51.749 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.108873.log 2023-08-26T18:17:51.749 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster1-client.admin.108873.log: 87.5% -- replaced with /var/log/ceph/cluster1-client.admin.108873.log.gz 2023-08-26T18:17:51.749 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.129099.log 2023-08-26T18:17:51.750 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster2-client.mirror.129099.log: 89.8% -- replaced with /var/log/ceph/cluster2-client.mirror.129099.log.gz 2023-08-26T18:17:51.751 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.129375.log 2023-08-26T18:17:51.751 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster1-client.mirror.129375.log: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.129375.log.gz 2023-08-26T18:17:51.751 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.122427.log 2023-08-26T18:17:51.752 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster1-client.mirror.122427.log: 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.122427.log.gz 2023-08-26T18:17:51.753 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.126738.log 2023-08-26T18:17:51.753 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster1-client.mirror.126738.log: 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.126738.log.gz 2023-08-26T18:17:51.753 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.133776.log 2023-08-26T18:17:51.754 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster2-client.mirror.133776.log: 87.9% -- replaced with /var/log/ceph/cluster2-client.mirror.133776.log.gz 2023-08-26T18:17:51.755 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.133528.log 2023-08-26T18:17:51.755 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster1-client.mirror.133528.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.133528.log.gz 2023-08-26T18:17:51.755 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.134346.log 2023-08-26T18:17:51.755 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster2-client.mirror.134346.log: 88.0% -- replaced with /var/log/ceph/cluster2-client.mirror.134346.log.gz 2023-08-26T18:17:51.756 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.120357.log 2023-08-26T18:17:51.756 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster2-client.mirror.120357.log: 87.4% -- replaced with /var/log/ceph/cluster2-client.mirror.120357.log.gz 2023-08-26T18:17:51.756 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.142895.log 2023-08-26T18:17:51.757 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster1-client.mirror.142895.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.142895.log.gz 2023-08-26T18:17:51.757 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.122599.log 2023-08-26T18:17:51.757 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster1-client.mirror.122599.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.122599.log.gz 2023-08-26T18:17:51.757 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.135070.log 2023-08-26T18:17:51.758 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster2-client.mirror.135070.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.115181.log 2023-08-26T18:17:51.758 INFO:teuthology.orchestra.run.smithi124.stderr: 90.5% -- replaced with /var/log/ceph/cluster2-client.mirror.135070.log.gz 2023-08-26T18:17:51.767 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster2-client.mirror.115181.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.133941.log 2023-08-26T18:17:51.780 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster2-client.mirror.133941.log: 90.5% -- replaced with /var/log/ceph/cluster2-client.mirror.133941.log.gz 2023-08-26T18:17:51.781 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.122490.log 2023-08-26T18:17:51.792 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster2-client.mirror.122490.log: 89.8% -- replaced with /var/log/ceph/cluster2-client.mirror.122490.log.gz 2023-08-26T18:17:51.793 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.124116.log 2023-08-26T18:17:51.806 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster1-client.mirror.124116.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.116971.log 2023-08-26T18:17:51.817 INFO:teuthology.orchestra.run.smithi124.stderr: 92.3% -- replaced with /var/log/ceph/cluster1-client.mirror.124116.log.gz 2023-08-26T18:17:51.828 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster2-client.mirror.116971.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.142822.log 2023-08-26T18:17:51.848 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster1-client.mirror.142822.log: 48.2% -- replaced with /var/log/ceph/cluster1-client.mirror.142822.log.gz 2023-08-26T18:17:51.848 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.136643.log 2023-08-26T18:17:51.850 INFO:teuthology.orchestra.run.smithi124.stderr: 93.7% -- replaced with /var/log/ceph/cluster2-client.mirror.109880.log.gz 2023-08-26T18:17:51.853 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster2-client.mirror.136643.log: 90.4% -- replaced with /var/log/ceph/cluster2-client.mirror.136643.log.gz 2023-08-26T18:17:51.853 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.112801.log 2023-08-26T18:17:51.866 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster1-client.admin.112801.log: 88.2% -- replaced with /var/log/ceph/cluster1-client.admin.112801.log.gz 2023-08-26T18:17:51.866 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.125720.log 2023-08-26T18:17:51.880 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.116864.log 2023-08-26T18:17:51.892 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster2-client.mirror.125720.log: 92.4% -- replaced with /var/log/ceph/cluster2-client.mirror.125720.log.gz 2023-08-26T18:17:51.906 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster2-client.mirror.116864.log: 79.6% -- replaced with /var/log/ceph/cluster2-client.mirror.116864.log.gz 2023-08-26T18:17:51.906 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.118968.log 2023-08-26T18:17:51.915 INFO:teuthology.orchestra.run.smithi124.stderr: 93.7% -- replaced with /var/log/ceph/cluster2-client.mirror.116389.log.gz 2023-08-26T18:17:51.920 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster2-client.mirror.118968.log: 89.6% -- replaced with /var/log/ceph/cluster2-client.mirror.118968.log.gz 2023-08-26T18:17:51.920 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.0.133344.log 2023-08-26T18:17:51.921 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster1-client.mirror.0.133344.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.141508.log 2023-08-26T18:17:51.933 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster1-client.mirror.141508.log: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.141508.log.gz 2023-08-26T18:17:51.933 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.135360.log 2023-08-26T18:17:51.944 INFO:teuthology.orchestra.run.smithi124.stderr: 93.7% -- replaced with /var/log/ceph/cluster2-client.mirror.115181.log.gz 2023-08-26T18:17:51.944 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster2-client.mirror.135360.log: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.135360.log.gz 2023-08-26T18:17:51.945 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.126979.log 2023-08-26T18:17:51.945 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster2-client.mirror.126979.log: 92.6% -- replaced with /var/log/ceph/cluster2-client.mirror.126979.log.gz 2023-08-26T18:17:51.945 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.132354.log 2023-08-26T18:17:51.946 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster1-client.mirror.132354.log: 92.2% -- replaced with /var/log/ceph/cluster1-client.mirror.132354.log.gz 2023-08-26T18:17:51.946 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.109704.log 2023-08-26T18:17:51.946 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster1-client.mirror.109704.log: 88.1% -- replaced with /var/log/ceph/cluster1-client.mirror.109704.log.gz 2023-08-26T18:17:51.946 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.116690.log 2023-08-26T18:17:51.947 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster2-client.mirror.116690.log: 89.6% -- replaced with /var/log/ceph/cluster2-client.mirror.116690.log.gz 2023-08-26T18:17:51.947 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.128635.log 2023-08-26T18:17:51.947 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster2-client.mirror.128635.log: 89.9% -- replaced with /var/log/ceph/cluster2-client.mirror.128635.log.gz 2023-08-26T18:17:51.947 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.112378.log 2023-08-26T18:17:51.948 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster1-client.mirror.112378.log: 88.4% -- replaced with /var/log/ceph/cluster1-client.mirror.112378.log.gz 2023-08-26T18:17:51.948 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.109094.log 2023-08-26T18:17:51.949 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster2-client.mirror.109094.log: 90.0% -- replaced with /var/log/ceph/cluster2-client.mirror.109094.log.gz 2023-08-26T18:17:51.949 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.132747.log 2023-08-26T18:17:51.950 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster1-client.mirror.132747.log: 87.2% -- replaced with /var/log/ceph/cluster1-client.mirror.132747.log.gz 2023-08-26T18:17:51.950 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.125901.log 2023-08-26T18:17:51.950 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster1-client.mirror.125901.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.125901.log.gz 2023-08-26T18:17:51.951 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.144528.log 2023-08-26T18:17:51.951 INFO:teuthology.orchestra.run.smithi124.stderr: 91.9% -- replaced with /var/log/ceph/cluster1-client.mirror.0.133344.log.gz 2023-08-26T18:17:51.951 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster2-client.mirror.144528.log: 8.2% -- replaced with /var/log/ceph/cluster2-client.mirror.144528.log.gz 2023-08-26T18:17:51.952 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.128292.log 2023-08-26T18:17:51.956 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster2-client.mirror.128292.log: 93.8% -- replaced with /var/log/ceph/cluster2-client.mirror.116971.log.gz 2023-08-26T18:17:51.956 INFO:teuthology.orchestra.run.smithi124.stderr: 92.4% -- replaced with /var/log/ceph/cluster2-client.mirror.128292.log.gz 2023-08-26T18:17:51.957 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.134988.log 2023-08-26T18:17:51.957 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.142091.log 2023-08-26T18:17:51.957 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster2-client.mirror.134988.log: /var/log/ceph/cluster2-client.mirror.142091.log: 88.0% -- replaced with /var/log/ceph/cluster2-client.mirror.134988.log.gz 2023-08-26T18:17:51.958 INFO:teuthology.orchestra.run.smithi124.stderr: 79.7% -- replaced with /var/log/ceph/cluster2-client.mirror.142091.log.gz 2023-08-26T18:17:51.958 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.136354.log 2023-08-26T18:17:51.958 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.132289.log 2023-08-26T18:17:51.959 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster2-client.mirror.136354.log: /var/log/ceph/cluster1-client.mirror.132289.log: 87.9% -- replaced with /var/log/ceph/cluster2-client.mirror.136354.log.gz 2023-08-26T18:17:51.959 INFO:teuthology.orchestra.run.smithi124.stderr: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.132289.log.gz 2023-08-26T18:17:51.959 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.114497.log 2023-08-26T18:17:51.960 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.128151.log 2023-08-26T18:17:51.960 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster2-client.mirror.114497.log: /var/log/ceph/cluster2-client.mirror.128151.log: 88.6% -- replaced with /var/log/ceph/cluster2-client.mirror.114497.log.gz 2023-08-26T18:17:51.960 INFO:teuthology.orchestra.run.smithi124.stderr: 87.3% -- replaced with /var/log/ceph/cluster2-client.mirror.128151.log.gz 2023-08-26T18:17:51.960 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.145210.log 2023-08-26T18:17:51.961 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.125853.log 2023-08-26T18:17:51.961 INFO:teuthology.orchestra.run.smithi124.stderr: 93.7% -- replaced with /var/log/ceph/cluster2-client.mirror.119763.log.gz 2023-08-26T18:17:51.962 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster2-client.mirror.145210.log: 8.2% -- replaced with /var/log/ceph/cluster2-client.mirror.145210.log.gzgzip 2023-08-26T18:17:51.962 INFO:teuthology.orchestra.run.smithi124.stderr:-5 --verbose -- /var/log/ceph/cluster1-client.mirror.130545.log 2023-08-26T18:17:51.962 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.134024.log 2023-08-26T18:17:51.963 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster1-client.mirror.125853.log: /var/log/ceph/cluster1-client.mirror.130545.log: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.125853.log.gz 2023-08-26T18:17:51.963 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster2-client.mirror.134024.log: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.130545.log.gz 2023-08-26T18:17:51.964 INFO:teuthology.orchestra.run.smithi124.stderr: 87.9% -- replaced with /var/log/ceph/cluster2-client.mirror.134024.log.gz 2023-08-26T18:17:51.964 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.124577.log 2023-08-26T18:17:51.970 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.3.106124.log 2023-08-26T18:17:51.970 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster1-client.mirror.124577.log: /var/log/ceph/cluster1-client.mirror.3.106124.log: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.124577.log.gz 2023-08-26T18:17:51.971 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.116631.log 2023-08-26T18:17:51.971 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.125157.log 2023-08-26T18:17:51.971 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster2-client.mirror.116631.log: /var/log/ceph/cluster1-client.mirror.125157.log: 87.5% -- replaced with /var/log/ceph/cluster2-client.mirror.116631.log.gz 2023-08-26T18:17:51.972 INFO:teuthology.orchestra.run.smithi124.stderr: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.125157.log.gz 2023-08-26T18:17:51.972 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.110534.log 2023-08-26T18:17:51.972 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.135920.log 2023-08-26T18:17:51.973 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster1-client.admin.110534.log: /var/log/ceph/cluster2-client.mirror.135920.log: 87.8% -- replaced with /var/log/ceph/cluster1-client.admin.110534.log.gz 2023-08-26T18:17:51.973 INFO:teuthology.orchestra.run.smithi124.stderr: 88.1% -- replaced with /var/log/ceph/cluster2-client.mirror.135920.log.gz 2023-08-26T18:17:51.973 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.140359.log 2023-08-26T18:17:51.974 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.123414.log 2023-08-26T18:17:51.974 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster1-client.mirror.140359.log: 86.8% -- replaced with /var/log/ceph/cluster1-client.mirror.140359.log.gz 2023-08-26T18:17:51.974 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster1-client.mirror.123414.log: 92.3% -- replaced with /var/log/ceph/cluster1-client.mirror.123414.log.gz 2023-08-26T18:17:51.974 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.120711.log 2023-08-26T18:17:51.975 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.136416.log 2023-08-26T18:17:51.975 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster1-client.mirror.120711.log: /var/log/ceph/cluster2-client.mirror.136416.log: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.120711.log.gz 2023-08-26T18:17:51.975 INFO:teuthology.orchestra.run.smithi124.stderr: 88.0% -- replaced with /var/log/ceph/cluster2-client.mirror.136416.log.gz 2023-08-26T18:17:51.975 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.120967.log 2023-08-26T18:17:51.976 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.117475.log 2023-08-26T18:17:51.976 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster1-client.mirror.120967.log: /var/log/ceph/cluster2-client.mirror.117475.log: 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.120967.log.gz 2023-08-26T18:17:51.976 INFO:teuthology.orchestra.run.smithi124.stderr: 89.7% -- replaced with /var/log/ceph/cluster2-client.mirror.117475.log.gz 2023-08-26T18:17:51.976 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.117191.log 2023-08-26T18:17:51.977 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.110828.log 2023-08-26T18:17:51.977 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster2-client.mirror.117191.log: 79.5% -- replaced with /var/log/ceph/cluster2-client.mirror.117191.log.gz 2023-08-26T18:17:51.977 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster1-client.mirror.110828.log: 88.5% -- replaced with /var/log/ceph/cluster1-client.mirror.110828.log.gz 2023-08-26T18:17:51.978 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.116104.log 2023-08-26T18:17:51.978 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.106711.log 2023-08-26T18:17:51.978 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster2-client.mirror.116104.log: /var/log/ceph/cluster1-client.mirror.106711.log: 87.4% -- replaced with /var/log/ceph/cluster2-client.mirror.116104.log.gz 2023-08-26T18:17:51.978 INFO:teuthology.orchestra.run.smithi124.stderr: 54.5% -- replaced with /var/log/ceph/cluster1-client.mirror.106711.log.gz 2023-08-26T18:17:51.979 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.108345.log 2023-08-26T18:17:51.980 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.142363.log 2023-08-26T18:17:51.980 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster1-client.mirror.108345.log: /var/log/ceph/cluster1-client.mirror.142363.log: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.108345.log.gz 2023-08-26T18:17:51.980 INFO:teuthology.orchestra.run.smithi124.stderr: 86.8% -- replaced with /var/log/ceph/cluster1-client.mirror.142363.log.gz 2023-08-26T18:17:51.981 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.109816.log 2023-08-26T18:17:51.981 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.128334.log 2023-08-26T18:17:51.983 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster2-client.mirror.109816.log: /var/log/ceph/cluster2-client.mirror.128334.log: 89.8% -- replaced with /var/log/ceph/cluster2-client.mirror.128334.log.gz 2023-08-26T18:17:51.983 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.118755.log 2023-08-26T18:17:51.983 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster2-client.mirror.118755.log: 93.4% -- replaced with /var/log/ceph/cluster2-client.mirror.109816.log.gz 2023-08-26T18:17:51.984 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.134719.log 2023-08-26T18:17:51.984 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.138770.log 2023-08-26T18:17:51.984 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster2-client.mirror.134719.log: 88.0% -- replaced with /var/log/ceph/cluster2-client.mirror.134719.log.gz 2023-08-26T18:17:51.987 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster1-client.mirror.138770.log: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.138770.log.gz 2023-08-26T18:17:51.987 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.120687.log 2023-08-26T18:17:51.987 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.124971.log 2023-08-26T18:17:51.988 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster1-client.mirror.120687.log: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.120687.log.gz 2023-08-26T18:17:51.988 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster1-client.mirror.124971.log: 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.124971.log.gz 2023-08-26T18:17:51.988 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.107852.log 2023-08-26T18:17:51.989 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster1-client.admin.107852.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.admin.107852.log.gz 2023-08-26T18:17:51.989 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.111278.log 2023-08-26T18:17:51.989 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.119373.log 2023-08-26T18:17:51.992 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster2-client.mirror.111278.log: 90.0% -- replaced with /var/log/ceph/cluster2-client.mirror.111278.log.gz 2023-08-26T18:17:51.999 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster2-client.mirror.119373.log: 89.6% -- replaced with /var/log/ceph/cluster2-client.mirror.119373.log.gz 2023-08-26T18:17:51.999 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.141641.log 2023-08-26T18:17:52.000 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.135236.log 2023-08-26T18:17:52.001 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.105193.log 2023-08-26T18:17:52.001 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster2-client.mirror.135236.log: /var/log/ceph/cluster2-client.mirror.141641.log: 80.1% -- replaced with /var/log/ceph/cluster2-client.mirror.141641.log.gz 2023-08-26T18:17:52.001 INFO:teuthology.orchestra.run.smithi124.stderr: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.135236.log.gz 2023-08-26T18:17:52.011 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster2-client.admin.105193.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.105193.log.gz 2023-08-26T18:17:52.012 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.140733.log 2023-08-26T18:17:52.013 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster2-client.mirror.140733.log: 79.4% -- replaced with /var/log/ceph/cluster2-client.mirror.140733.log.gz 2023-08-26T18:17:52.013 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.119188.log 2023-08-26T18:17:52.013 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster2-client.mirror.119188.log: 79.5% -- replaced with /var/log/ceph/cluster2-client.mirror.119188.log.gz 2023-08-26T18:17:52.014 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.141131.log 2023-08-26T18:17:52.015 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster2-client.mirror.141131.log: 87.4% -- replaced with /var/log/ceph/cluster2-client.mirror.141131.log.gz 2023-08-26T18:17:52.015 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.117703.log 2023-08-26T18:17:52.016 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.143696.log 2023-08-26T18:17:52.016 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster2-client.mirror.117703.log: 79.5% -- replaced with /var/log/ceph/cluster2-client.mirror.117703.log.gz 2023-08-26T18:17:52.016 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster2-client.mirror.143696.log: 8.2% -- replaced with /var/log/ceph/cluster2-client.mirror.143696.log.gz 2023-08-26T18:17:52.017 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.130026.log 2023-08-26T18:17:52.018 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.125762.log 2023-08-26T18:17:52.019 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster1-client.mirror.130026.log: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.130026.log.gz 2023-08-26T18:17:52.019 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster2-client.mirror.125762.log: 89.8% -- replaced with /var/log/ceph/cluster2-client.mirror.125762.log.gz 2023-08-26T18:17:52.019 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.145673.log 2023-08-26T18:17:52.019 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.115641.log 2023-08-26T18:17:52.020 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster2-client.admin.145673.log: 8.2% -- replaced with /var/log/ceph/cluster2-client.admin.145673.log.gz 2023-08-26T18:17:52.020 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.138028.log 2023-08-26T18:17:52.021 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster2-client.mirror.115641.log: 87.4% -- replaced with /var/log/ceph/cluster2-client.mirror.115641.log.gz 2023-08-26T18:17:52.022 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster1-client.mirror.138028.log: 86.8% -- replaced with /var/log/ceph/cluster1-client.mirror.138028.log.gz 2023-08-26T18:17:52.023 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.121502.log 2023-08-26T18:17:52.024 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster1-client.mirror.121502.log: 92.3% -- replaced with /var/log/ceph/cluster1-client.mirror.121502.log.gz 2023-08-26T18:17:52.025 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.122115.log 2023-08-26T18:17:52.026 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster1-client.mirror.122115.log: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.122115.log.gz 2023-08-26T18:17:52.026 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.130345.log 2023-08-26T18:17:52.027 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster2-client.mirror.130345.log: 89.7% -- replaced with /var/log/ceph/cluster2-client.mirror.130345.log.gz 2023-08-26T18:17:52.027 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.137038.log 2023-08-26T18:17:52.028 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster1-client.mirror.137038.log: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.137038.log.gz 2023-08-26T18:17:52.028 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.99894.log 2023-08-26T18:17:52.029 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster2-client.admin.99894.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.99894.log.gz 2023-08-26T18:17:52.030 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.137690.log 2023-08-26T18:17:52.030 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster1-client.mirror.137690.log: 86.8% -- replaced with /var/log/ceph/cluster1-client.mirror.137690.log.gz 2023-08-26T18:17:52.031 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.118820.log 2023-08-26T18:17:52.033 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster2-client.mirror.118820.log: 87.4% -- replaced with /var/log/ceph/cluster2-client.mirror.118820.log.gz 2023-08-26T18:17:52.033 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.138838.log 2023-08-26T18:17:52.034 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster1-client.mirror.138838.log: 86.7% -- replaced with /var/log/ceph/cluster1-client.mirror.138838.log.gz 2023-08-26T18:17:52.034 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.113577.log 2023-08-26T18:17:52.038 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster1-client.mirror.113577.log: 93.3% -- replaced with /var/log/ceph/cluster1-client.mirror.113577.log.gz 2023-08-26T18:17:52.038 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.140510.log 2023-08-26T18:17:52.039 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster2-client.mirror.140510.log: 87.5% -- replaced with /var/log/ceph/cluster2-client.mirror.140510.log.gz 2023-08-26T18:17:52.039 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.117396.log 2023-08-26T18:17:52.040 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster2-client.mirror.117396.log: 89.6% -- replaced with /var/log/ceph/cluster2-client.mirror.117396.log.gz 2023-08-26T18:17:52.040 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.133817.log 2023-08-26T18:17:52.043 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster2-client.mirror.133817.log: 90.4% -- replaced with /var/log/ceph/cluster2-client.mirror.133817.log.gz 2023-08-26T18:17:52.043 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.131971.log 2023-08-26T18:17:52.044 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster1-client.mirror.131971.log: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.131971.log.gz 2023-08-26T18:17:52.044 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.133145.log 2023-08-26T18:17:52.045 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster1-client.mirror.133145.log: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.133145.log.gz 2023-08-26T18:17:52.045 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.118430.log 2023-08-26T18:17:52.046 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster2-client.mirror.118430.log: 79.6% -- replaced with /var/log/ceph/cluster2-client.mirror.118430.log.gz 2023-08-26T18:17:52.046 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.129447.log 2023-08-26T18:17:52.047 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster1-client.mirror.129447.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.129447.log.gz 2023-08-26T18:17:52.047 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.110763.log 2023-08-26T18:17:52.048 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster2-client.mirror.110763.log: 88.4% -- replaced with /var/log/ceph/cluster2-client.mirror.110763.log.gz 2023-08-26T18:17:52.048 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.107665.log 2023-08-26T18:17:52.049 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster1-client.admin.107665.log: 87.3% -- replaced with /var/log/ceph/cluster1-client.admin.107665.log.gz 2023-08-26T18:17:52.049 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.126598.log 2023-08-26T18:17:52.050 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster2-client.mirror.126598.log: 89.8% -- replaced with /var/log/ceph/cluster2-client.mirror.126598.log.gz 2023-08-26T18:17:52.050 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.142202.log 2023-08-26T18:17:52.051 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster1-client.mirror.142202.log: 86.8% -- replaced with /var/log/ceph/cluster1-client.mirror.142202.log.gz 2023-08-26T18:17:52.051 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.133625.log 2023-08-26T18:17:52.053 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster1-client.mirror.133625.log: 92.3% -- replaced with /var/log/ceph/cluster1-client.mirror.133625.log.gz 2023-08-26T18:17:52.053 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.110900.log 2023-08-26T18:17:52.054 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster1-client.mirror.110900.log: 88.1% -- replaced with /var/log/ceph/cluster1-client.mirror.110900.log.gz 2023-08-26T18:17:52.054 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.126618.log 2023-08-26T18:17:52.055 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster2-client.mirror.126618.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.133304.log 2023-08-26T18:17:52.055 INFO:teuthology.orchestra.run.smithi124.stderr: 87.3% -- replaced with /var/log/ceph/cluster2-client.mirror.126618.log.gz 2023-08-26T18:17:52.056 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster2-client.mirror.133304.log: 89.7% -- replaced with /var/log/ceph/cluster2-client.mirror.133304.log.gz 2023-08-26T18:17:52.057 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.118783.log 2023-08-26T18:17:52.057 INFO:teuthology.orchestra.run.smithi124.stderr: 93.1% -- replaced with /var/log/ceph/cluster1-client.mirror.3.106124.log.gz 2023-08-26T18:17:52.058 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.130521.log 2023-08-26T18:17:52.059 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster2-client.mirror.118783.log: 79.6% -- replaced with /var/log/ceph/cluster2-client.mirror.118783.log.gz 2023-08-26T18:17:52.060 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster2-client.mirror.130521.log: 87.4% -- replaced with /var/log/ceph/cluster2-client.mirror.130521.log.gz 2023-08-26T18:17:52.060 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.116330.log 2023-08-26T18:17:52.060 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.144424.log 2023-08-26T18:17:52.061 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster2-client.mirror.116330.log: 79.7% -- replaced with /var/log/ceph/cluster2-client.mirror.116330.log.gz 2023-08-26T18:17:52.061 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster2-client.admin.144424.log: 8.2% -- replaced with /var/log/ceph/cluster2-client.admin.144424.log.gz 2023-08-26T18:17:52.061 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.127888.log 2023-08-26T18:17:52.062 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.133879.log 2023-08-26T18:17:52.063 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster1-client.mirror.127888.log: /var/log/ceph/cluster2-client.mirror.133879.log: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.127888.log.gz 2023-08-26T18:17:52.063 INFO:teuthology.orchestra.run.smithi124.stderr: 90.4% -- replaced with /var/log/ceph/cluster2-client.mirror.133879.log.gz 2023-08-26T18:17:52.064 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.142916.log 2023-08-26T18:17:52.064 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster1-client.mirror.142916.log: 67.7% -- replaced with /var/log/ceph/cluster1-client.mirror.142916.log.gz 2023-08-26T18:17:52.065 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.1.110691.log 2023-08-26T18:17:52.065 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.143044.log 2023-08-26T18:17:52.066 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster2-client.mirror.1.110691.log: /var/log/ceph/cluster1-client.mirror.143044.log: 56.2% -- replaced with /var/log/ceph/cluster1-client.mirror.143044.log.gz 2023-08-26T18:17:52.066 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.122204.log 2023-08-26T18:17:52.067 INFO:teuthology.orchestra.run.smithi124.stderr: 92.0% -- replaced with /var/log/ceph/cluster2-client.mirror.1.110691.log.gz 2023-08-26T18:17:52.068 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.134197.log 2023-08-26T18:17:52.069 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster1-client.mirror.122204.log: /var/log/ceph/cluster2-client.mirror.134197.log: 92.2% -- replaced with /var/log/ceph/cluster1-client.mirror.122204.log.gz 2023-08-26T18:17:52.070 INFO:teuthology.orchestra.run.smithi124.stderr: 90.5% -- replaced with /var/log/ceph/cluster2-client.mirror.134197.log.gz 2023-08-26T18:17:52.070 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.133193.log 2023-08-26T18:17:52.071 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.124315.log 2023-08-26T18:17:52.071 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster1-client.mirror.133193.log: /var/log/ceph/cluster1-client.mirror.124315.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.133193.log.gz 2023-08-26T18:17:52.072 INFO:teuthology.orchestra.run.smithi124.stderr: 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.124315.log.gz 2023-08-26T18:17:52.072 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.131226.log 2023-08-26T18:17:52.073 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.131374.log 2023-08-26T18:17:52.074 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster1-client.mirror.131226.log: /var/log/ceph/cluster2-client.mirror.131374.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.131226.log.gz 2023-08-26T18:17:52.074 INFO:teuthology.orchestra.run.smithi124.stderr: 87.4% -- replaced with /var/log/ceph/cluster2-client.mirror.131374.log.gz 2023-08-26T18:17:52.074 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.140704.log 2023-08-26T18:17:52.074 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.122468.log 2023-08-26T18:17:52.075 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster2-client.mirror.140704.log: /var/log/ceph/cluster1-client.mirror.122468.log: 92.3% -- replaced with /var/log/ceph/cluster1-client.mirror.122468.log.gz 2023-08-26T18:17:52.076 INFO:teuthology.orchestra.run.smithi124.stderr: 91.6% -- replaced with /var/log/ceph/cluster2-client.mirror.140704.log.gz 2023-08-26T18:17:52.076 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.108528.log 2023-08-26T18:17:52.076 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.139104.log 2023-08-26T18:17:52.077 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster1-client.mirror.108528.log: /var/log/ceph/cluster2-client.mirror.139104.log: 87.6% -- replaced with /var/log/ceph/cluster1-client.mirror.108528.log.gz 2023-08-26T18:17:52.078 INFO:teuthology.orchestra.run.smithi124.stderr: 1.3% -- replaced with /var/log/ceph/cluster2-client.mirror.139104.log.gz 2023-08-26T18:17:52.078 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.0.109490.log 2023-08-26T18:17:52.078 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.118214.log 2023-08-26T18:17:52.079 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster1-client.mirror.0.109490.log: /var/log/ceph/cluster2-client.mirror.118214.log: 91.9% -- replaced with /var/log/ceph/cluster1-client.mirror.0.109490.log.gz 2023-08-26T18:17:52.086 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.134408.log 2023-08-26T18:17:52.086 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster2-client.mirror.134408.log: 88.0% -- replaced with /var/log/ceph/cluster2-client.mirror.134408.log.gz 2023-08-26T18:17:52.087 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.130002.log 2023-08-26T18:17:52.087 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster1-client.mirror.130002.log: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.130002.log.gz 2023-08-26T18:17:52.087 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.123349.log 2023-08-26T18:17:52.088 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster1-client.mirror.123349.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.123349.log.gz 2023-08-26T18:17:52.088 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.114245.log 2023-08-26T18:17:52.090 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster2-client.mirror.114245.log: 93.4% -- replaced with /var/log/ceph/cluster2-client.mirror.114245.log.gz 2023-08-26T18:17:52.090 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.122139.log 2023-08-26T18:17:52.091 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster1-client.mirror.122139.log: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.122139.log.gz 2023-08-26T18:17:52.092 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.127021.log 2023-08-26T18:17:52.092 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster2-client.mirror.127021.log: 89.7% -- replaced with /var/log/ceph/cluster2-client.mirror.127021.log.gz 2023-08-26T18:17:52.093 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.111322.log 2023-08-26T18:17:52.095 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster1-client.mirror.111322.log: 88.5% -- replaced with /var/log/ceph/cluster1-client.mirror.111322.log.gz 2023-08-26T18:17:52.095 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.123138.log 2023-08-26T18:17:52.096 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster2-client.mirror.123138.log: 92.4% -- replaced with /var/log/ceph/cluster2-client.mirror.123138.log.gz 2023-08-26T18:17:52.097 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.120284.log 2023-08-26T18:17:52.097 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster2-client.mirror.120284.log: 89.8% -- replaced with /var/log/ceph/cluster2-client.mirror.120284.log.gz 2023-08-26T18:17:52.098 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.129167.log 2023-08-26T18:17:52.099 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster1-client.mirror.129167.log: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.129167.log.gz 2023-08-26T18:17:52.100 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.132675.log 2023-08-26T18:17:52.100 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster2-client.mirror.132675.log: 87.3% -- replaced with /var/log/ceph/cluster2-client.mirror.132675.log.gz 2023-08-26T18:17:52.100 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.127457.log 2023-08-26T18:17:52.101 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster2-client.mirror.127457.log: 92.6% -- replaced with /var/log/ceph/cluster2-client.mirror.127457.log.gz 2023-08-26T18:17:52.101 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.133504.log 2023-08-26T18:17:52.102 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster1-client.mirror.133504.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.133504.log.gz 2023-08-26T18:17:52.103 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.123093.log 2023-08-26T18:17:52.103 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster1-client.mirror.123093.log: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.123093.log.gz 2023-08-26T18:17:52.104 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.108552.log 2023-08-26T18:17:52.104 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster1-client.mirror.108552.log: 87.8% -- replaced with /var/log/ceph/cluster1-client.mirror.108552.log.gz 2023-08-26T18:17:52.105 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.110564.log 2023-08-26T18:17:52.106 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster1-client.mirror.110564.log: 87.6% -- replaced with /var/log/ceph/cluster1-client.mirror.110564.log.gz 2023-08-26T18:17:52.106 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.129679.log 2023-08-26T18:17:52.107 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster1-client.mirror.129679.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.129679.log.gz 2023-08-26T18:17:52.107 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.128374.log 2023-08-26T18:17:52.117 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster2-client.mirror.128374.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.133692.log 2023-08-26T18:17:52.124 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster2-client.mirror.133692.log: 90.5% -- replaced with /var/log/ceph/cluster2-client.mirror.133692.log.gz 2023-08-26T18:17:52.124 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.127816.log 2023-08-26T18:17:52.125 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster2-client.mirror.127816.log: 93.5% -- replaced with /var/log/ceph/cluster2-client.mirror.128374.log.gz 2023-08-26T18:17:52.125 INFO:teuthology.orchestra.run.smithi124.stderr: 87.4% -- replaced with /var/log/ceph/cluster2-client.mirror.127816.log.gz 2023-08-26T18:17:52.125 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.108025.log 2023-08-26T18:17:52.126 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.137060.log 2023-08-26T18:17:52.126 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster1-client.mirror.108025.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.108025.log.gz 2023-08-26T18:17:52.127 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster1-client.mirror.137060.log: 86.7% -- replaced with /var/log/ceph/cluster1-client.mirror.137060.log.gz 2023-08-26T18:17:52.127 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.122825.log 2023-08-26T18:17:52.129 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.144683.log 2023-08-26T18:17:52.129 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster1-client.mirror.122825.log: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.122825.log.gz 2023-08-26T18:17:52.129 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster2-client.mirror.144683.log: 6.1% -- replaced with /var/log/ceph/cluster2-client.mirror.144683.log.gz 2023-08-26T18:17:52.130 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.118546.log 2023-08-26T18:17:52.130 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.113772.log 2023-08-26T18:17:52.131 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster2-client.mirror.118546.log: 87.4% -- replaced with /var/log/ceph/cluster2-client.mirror.118546.log.gz 2023-08-26T18:17:52.133 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster2-client.mirror.113772.log: 93.4% -- replaced with /var/log/ceph/cluster2-client.mirror.113772.log.gz 2023-08-26T18:17:52.135 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.116347.log 2023-08-26T18:17:52.138 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster2-client.mirror.116347.log: 89.8% -- replaced with /var/log/ceph/cluster2-client.mirror.116347.log.gz 2023-08-26T18:17:52.138 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.145004.log 2023-08-26T18:17:52.138 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster2-client.mirror.145004.log: 8.2% -- replaced with /var/log/ceph/cluster2-client.mirror.145004.log.gz 2023-08-26T18:17:52.138 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.135548.log 2023-08-26T18:17:52.139 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster2-client.mirror.135548.log: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.135548.log.gz 2023-08-26T18:17:52.139 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.120405.log 2023-08-26T18:17:52.140 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster1-client.mirror.120405.log: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.120405.log.gz 2023-08-26T18:17:52.141 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.132479.log 2023-08-26T18:17:52.141 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster1-client.mirror.132479.log: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.132479.log.gz 2023-08-26T18:17:52.141 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.116300.log 2023-08-26T18:17:52.142 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster2-client.mirror.116300.log: 93.8% -- replaced with /var/log/ceph/cluster2-client.mirror.141153.log.gz 2023-08-26T18:17:52.143 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.135256.log 2023-08-26T18:17:52.145 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster2-client.mirror.135256.log: 90.4% -- replaced with /var/log/ceph/cluster2-client.mirror.135256.log.gz 2023-08-26T18:17:52.145 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.134259.log 2023-08-26T18:17:52.147 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster2-client.mirror.134259.log: 90.4% -- replaced with /var/log/ceph/cluster2-client.mirror.134259.log.gz 2023-08-26T18:17:52.148 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.116611.log 2023-08-26T18:17:52.148 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.110876.log 2023-08-26T18:17:52.149 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster2-client.mirror.116611.log: /var/log/ceph/cluster1-client.mirror.110876.log: 89.7% -- replaced with /var/log/ceph/cluster2-client.mirror.116611.log.gz 2023-08-26T18:17:52.149 INFO:teuthology.orchestra.run.smithi124.stderr: 88.4% -- replaced with /var/log/ceph/cluster1-client.mirror.110876.log.gz 2023-08-26T18:17:52.150 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.135486.log 2023-08-26T18:17:52.151 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster2-client.mirror.135486.log: 88.1% -- replaced with /var/log/ceph/cluster2-client.mirror.135486.log.gz 2023-08-26T18:17:52.151 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.144634.log 2023-08-26T18:17:52.156 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster2-client.mirror.144634.log: 6.1% -- replaced with /var/log/ceph/cluster2-client.mirror.144634.log.gz 2023-08-26T18:17:52.156 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.104914.log 2023-08-26T18:17:52.161 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster2-client.admin.104914.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.104914.log.gz 2023-08-26T18:17:52.161 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.142317.log 2023-08-26T18:17:52.162 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.130952.log 2023-08-26T18:17:52.162 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster1-client.mirror.142317.log: 86.8% -- replaced with /var/log/ceph/cluster1-client.mirror.142317.log.gz 2023-08-26T18:17:52.162 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster1-client.mirror.130952.log: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.130952.log.gz 2023-08-26T18:17:52.162 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.104824.log 2023-08-26T18:17:52.163 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.137870.log 2023-08-26T18:17:52.163 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster2-client.admin.104824.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.104824.log.gz 2023-08-26T18:17:52.163 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.114050.log 2023-08-26T18:17:52.164 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster1-client.mirror.137870.log: 86.8% -- replaced with /var/log/ceph/cluster1-client.mirror.137870.log.gz 2023-08-26T18:17:52.165 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster1-client.mirror.114050.log: 93.4% -- replaced with /var/log/ceph/cluster1-client.mirror.114050.log.gz 2023-08-26T18:17:52.165 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.140598.log 2023-08-26T18:17:52.166 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster2-client.mirror.140598.log: 88.5% -- replaced with /var/log/ceph/cluster2-client.mirror.140598.log.gz 2023-08-26T18:17:52.166 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.143190.log 2023-08-26T18:17:52.168 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.145530.log 2023-08-26T18:17:52.168 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster1-client.mirror.143190.log: 2.3% -- replaced with /var/log/ceph/cluster1-client.mirror.143190.log.gz 2023-08-26T18:17:52.169 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster2-client.mirror.145530.log: 8.2% -- replaced with /var/log/ceph/cluster2-client.mirror.145530.log.gz 2023-08-26T18:17:52.169 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.105468.log 2023-08-26T18:17:52.169 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster2-client.admin.105468.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.105468.log.gz 2023-08-26T18:17:52.170 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.123735.log 2023-08-26T18:17:52.170 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster2-client.mirror.123735.log: 87.4% -- replaced with /var/log/ceph/cluster2-client.mirror.123735.log.gz 2023-08-26T18:17:52.171 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.110222.log 2023-08-26T18:17:52.171 INFO:teuthology.orchestra.run.smithi124.stderr:gzip /var/log/ceph/cluster1-client.mirror.110222.log: -5 --verbose -- /var/log/ceph/cluster2-client.mirror.115511.log 2023-08-26T18:17:52.172 INFO:teuthology.orchestra.run.smithi124.stderr: 88.2% -- replaced with /var/log/ceph/cluster1-client.mirror.110222.log.gz 2023-08-26T18:17:52.174 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster2-client.mirror.115511.log: 79.7% -- replaced with /var/log/ceph/cluster2-client.mirror.115511.log.gz 2023-08-26T18:17:52.175 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.127499.log 2023-08-26T18:17:52.175 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster2-client.mirror.127499.log: 89.8% -- replaced with /var/log/ceph/cluster2-client.mirror.127499.log.gz 2023-08-26T18:17:52.176 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.137848.log 2023-08-26T18:17:52.176 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster1-client.mirror.137848.log: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.137848.log.gz 2023-08-26T18:17:52.177 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.126551.log 2023-08-26T18:17:52.178 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster2-client.mirror.126551.log: 89.6% -- replaced with /var/log/ceph/cluster2-client.mirror.126551.log.gz 2023-08-26T18:17:52.178 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.0.112897.log 2023-08-26T18:17:52.180 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster2-client.mirror.0.112897.log: 92.3% -- replaced with /var/log/ceph/cluster2-client.mirror.0.112897.log.gz 2023-08-26T18:17:52.181 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.115356.log 2023-08-26T18:17:52.184 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster2-client.mirror.115356.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.143973.log 2023-08-26T18:17:52.184 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster2-client.admin.143973.log: 8.2% -- replaced with /var/log/ceph/cluster2-client.admin.143973.log.gz 2023-08-26T18:17:52.185 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.114286.log 2023-08-26T18:17:52.187 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster2-client.mirror.114286.log: 93.6% -- replaced with /var/log/ceph/cluster2-client.mirror.114286.log.gz 2023-08-26T18:17:52.187 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.127715.log 2023-08-26T18:17:52.189 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster1-client.mirror.127715.log: 92.4% -- replaced with /var/log/ceph/cluster1-client.mirror.127715.log.gz 2023-08-26T18:17:52.190 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.109424.log 2023-08-26T18:17:52.191 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster1-client.mirror.109424.log: 87.6% -- replaced with /var/log/ceph/cluster1-client.mirror.109424.log.gz 2023-08-26T18:17:52.191 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.118155.log 2023-08-26T18:17:52.192 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster2-client.mirror.118155.log: 79.8% -- replaced with /var/log/ceph/cluster2-client.mirror.118155.log.gz 2023-08-26T18:17:52.192 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.117675.log 2023-08-26T18:17:52.196 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster2-client.mirror.117675.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.117917.log 2023-08-26T18:17:52.220 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster2-client.mirror.117917.log: 87.4% -- replaced with /var/log/ceph/cluster2-client.mirror.117917.log.gz 2023-08-26T18:17:52.221 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.133713.log 2023-08-26T18:17:52.233 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster2-client.mirror.133713.log: 88.0% -- replaced with /var/log/ceph/cluster2-client.mirror.133713.log.gz 2023-08-26T18:17:52.244 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.106173.log 2023-08-26T18:17:52.255 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster1-client.mirror.106173.log: 83.8% -- replaced with /var/log/ceph/cluster1-client.mirror.106173.log.gz 2023-08-26T18:17:52.255 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.141270.log 2023-08-26T18:17:52.278 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster1-client.mirror.141270.log: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.141270.log.gz 2023-08-26T18:17:52.278 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.111482.log 2023-08-26T18:17:52.296 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster2-client.mirror.111482.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.134656.log 2023-08-26T18:17:52.297 INFO:teuthology.orchestra.run.smithi124.stderr: 93.4% -- replaced with /var/log/ceph/cluster2-client.mirror.111482.log.gz 2023-08-26T18:17:52.304 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster2-client.mirror.134656.log: 88.0% -- replaced with /var/log/ceph/cluster2-client.mirror.134656.log.gz 2023-08-26T18:17:52.314 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.126714.log 2023-08-26T18:17:52.326 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster1-client.mirror.126714.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.126714.log.gz 2023-08-26T18:17:52.326 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.105146.log 2023-08-26T18:17:52.327 INFO:teuthology.orchestra.run.smithi124.stderr: 93.7% -- replaced with /var/log/ceph/cluster2-client.mirror.118214.log.gz 2023-08-26T18:17:52.327 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster2-client.admin.105146.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.105146.log.gz 2023-08-26T18:17:52.327 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.108781.log 2023-08-26T18:17:52.328 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster2-client.mirror.108781.log: 74.4% -- replaced with /var/log/ceph/cluster2-client.mirror.108781.log.gz 2023-08-26T18:17:52.328 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.127436.log 2023-08-26T18:17:52.329 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster1-client.mirror.127436.log: 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.127436.log.gz 2023-08-26T18:17:52.329 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.117537.log 2023-08-26T18:17:52.330 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster2-client.mirror.117537.log: 79.6% -- replaced with /var/log/ceph/cluster2-client.mirror.117537.log.gz 2023-08-26T18:17:52.330 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.135339.log 2023-08-26T18:17:52.331 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster2-client.mirror.135339.log: 88.0% -- replaced with /var/log/ceph/cluster2-client.mirror.135339.log.gz 2023-08-26T18:17:52.331 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.110174.log 2023-08-26T18:17:52.331 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster1-client.mirror.110174.log: 88.3% -- replaced with /var/log/ceph/cluster1-client.mirror.110174.log.gz 2023-08-26T18:17:52.332 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.116923.log 2023-08-26T18:17:52.340 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster2-client.mirror.116923.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.117653.log 2023-08-26T18:17:52.360 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster2-client.mirror.117653.log: 87.4% -- replaced with /var/log/ceph/cluster2-client.mirror.117653.log.gz 2023-08-26T18:17:52.362 INFO:teuthology.orchestra.run.smithi124.stderr: 93.7% -- replaced with /var/log/ceph/cluster2-client.mirror.117675.log.gz 2023-08-26T18:17:52.362 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.138793.log 2023-08-26T18:17:52.364 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster1-client.mirror.138793.log: 86.8% -- replaced with /var/log/ceph/cluster1-client.mirror.138793.log.gz 2023-08-26T18:17:52.365 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.108247.log 2023-08-26T18:17:52.371 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster1-client.mirror.108247.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.138500.log 2023-08-26T18:17:52.371 INFO:teuthology.orchestra.run.smithi124.stderr: 93.4% -- replaced with /var/log/ceph/cluster1-client.mirror.108247.log.gz 2023-08-26T18:17:52.372 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster1-client.mirror.138500.log: 86.8% -- replaced with /var/log/ceph/cluster1-client.mirror.138500.log.gz 2023-08-26T18:17:52.372 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.133755.log 2023-08-26T18:17:52.375 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster2-client.mirror.133755.log: 90.4% -- replaced with /var/log/ceph/cluster2-client.mirror.133755.log.gz 2023-08-26T18:17:52.375 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.126958.log 2023-08-26T18:17:52.376 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster1-client.mirror.126958.log: 87.6% -- replaced with /var/log/ceph/cluster1-client.mirror.126958.log.gz 2023-08-26T18:17:52.376 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.121151.log 2023-08-26T18:17:52.378 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster1-client.mirror.121151.log: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.121151.log.gz 2023-08-26T18:17:52.378 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.109612.log 2023-08-26T18:17:52.379 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster2-client.mirror.109612.log: 90.0% -- replaced with /var/log/ceph/cluster2-client.mirror.109612.log.gz 2023-08-26T18:17:52.380 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.122067.log 2023-08-26T18:17:52.389 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster2-client.mirror.122067.log: 87.3% -- replaced with /var/log/ceph/cluster2-client.mirror.122067.log.gz 2023-08-26T18:17:52.389 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.127975.log 2023-08-26T18:17:52.391 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster2-client.mirror.127975.log: 89.8% -- replaced with /var/log/ceph/cluster2-client.mirror.127975.log.gz 2023-08-26T18:17:52.391 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.125809.log 2023-08-26T18:17:52.393 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster2-client.mirror.125809.log: 89.8% -- replaced with /var/log/ceph/cluster2-client.mirror.125809.log.gz 2023-08-26T18:17:52.394 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.117077.log 2023-08-26T18:17:52.395 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster2-client.mirror.117077.log: 93.7% -- replaced with /var/log/ceph/cluster2-client.mirror.118755.log.gz 2023-08-26T18:17:52.395 INFO:teuthology.orchestra.run.smithi124.stderr: 89.7% -- replaced with /var/log/ceph/cluster2-client.mirror.117077.log.gz 2023-08-26T18:17:52.395 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.138860.log 2023-08-26T18:17:52.396 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.141111.log 2023-08-26T18:17:52.396 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster1-client.mirror.138860.log: 86.8% -- replaced with /var/log/ceph/cluster1-client.mirror.138860.log.gz 2023-08-26T18:17:52.397 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster2-client.mirror.141111.log: 89.7% -- replaced with /var/log/ceph/cluster2-client.mirror.141111.log.gz 2023-08-26T18:17:52.397 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.115848.log 2023-08-26T18:17:52.398 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster2-client.mirror.115848.log: 87.4% -- replaced with /var/log/ceph/cluster2-client.mirror.115848.log.gz 2023-08-26T18:17:52.399 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.134615.log 2023-08-26T18:17:52.399 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster2-client.mirror.134615.log: 88.1% -- replaced with /var/log/ceph/cluster2-client.mirror.134615.log.gz 2023-08-26T18:17:52.399 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.128703.log 2023-08-26T18:17:52.400 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster1-client.mirror.128703.log: 87.5% -- replaced with /var/log/ceph/cluster1-client.mirror.128703.log.gz 2023-08-26T18:17:52.401 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.131178.log 2023-08-26T18:17:52.401 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster1-client.mirror.131178.log: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.131178.log.gz 2023-08-26T18:17:52.401 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.113358.log 2023-08-26T18:17:52.405 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster2-client.mirror.113358.log: 89.3% -- replaced with /var/log/ceph/cluster2-client.mirror.113358.log.gz 2023-08-26T18:17:52.405 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.99548.log 2023-08-26T18:17:52.406 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster2-client.admin.99548.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.99548.log.gz 2023-08-26T18:17:52.406 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.140770.log 2023-08-26T18:17:52.406 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster2-client.mirror.140770.log: 87.4% -- replaced with /var/log/ceph/cluster2-client.mirror.140770.log.gz 2023-08-26T18:17:52.407 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.135174.log 2023-08-26T18:17:52.407 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster2-client.mirror.135174.log: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.135174.log.gz 2023-08-26T18:17:52.407 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.138568.log 2023-08-26T18:17:52.408 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster1-client.mirror.138568.log: 86.8% -- replaced with /var/log/ceph/cluster1-client.mirror.138568.log.gz 2023-08-26T18:17:52.408 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.112022.log 2023-08-26T18:17:52.411 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster1-client.mirror.112022.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.122323.log 2023-08-26T18:17:52.412 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster1-client.mirror.122323.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.122323.log.gz 2023-08-26T18:17:52.412 INFO:teuthology.orchestra.run.smithi124.stderr: 93.4% -- replaced with /var/log/ceph/cluster1-client.mirror.112022.log.gz 2023-08-26T18:17:52.413 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.120261.log 2023-08-26T18:17:52.413 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster1-client.mirror.120261.log: 1.3% -- replaced with /var/log/ceph/cluster1-client.mirror.120261.log.gz 2023-08-26T18:17:52.413 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.120461.log 2023-08-26T18:17:52.416 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster1-client.mirror.120461.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.120461.log.gz 2023-08-26T18:17:52.416 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.116789.log 2023-08-26T18:17:52.417 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster2-client.mirror.116789.log: 87.5% -- replaced with /var/log/ceph/cluster2-client.mirror.116789.log.gz 2023-08-26T18:17:52.417 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.141174.log 2023-08-26T18:17:52.417 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster1-client.mirror.141174.log: 87.2% -- replaced with /var/log/ceph/cluster1-client.mirror.141174.log.gz 2023-08-26T18:17:52.418 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.137510.log 2023-08-26T18:17:52.418 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster1-client.mirror.137510.log: 86.7% -- replaced with /var/log/ceph/cluster1-client.mirror.137510.log.gz 2023-08-26T18:17:52.418 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.143859.log 2023-08-26T18:17:52.419 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.139139.log 2023-08-26T18:17:52.419 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster2-client.mirror.143859.log: 8.2% -- replaced with /var/log/ceph/cluster2-client.mirror.143859.log.gz 2023-08-26T18:17:52.420 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster2-client.mirror.139139.log: 89.7% -- replaced with /var/log/ceph/cluster2-client.mirror.139139.log.gz 2023-08-26T18:17:52.420 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.1.110008.log 2023-08-26T18:17:52.421 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster2-client.mirror.1.110008.log: 91.7% -- replaced with /var/log/ceph/cluster2-client.mirror.1.110008.log.gz 2023-08-26T18:17:52.421 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.137353.log 2023-08-26T18:17:52.423 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.123436.log 2023-08-26T18:17:52.424 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster1-client.mirror.137353.log: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.137353.log.gz 2023-08-26T18:17:52.425 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster2-client.mirror.123436.log: 89.6% -- replaced with /var/log/ceph/cluster2-client.mirror.123436.log.gz 2023-08-26T18:17:52.425 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.142225.log 2023-08-26T18:17:52.426 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.124947.log 2023-08-26T18:17:52.426 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster1-client.mirror.142225.log: 86.7% -- replaced with /var/log/ceph/cluster1-client.mirror.142225.log.gz 2023-08-26T18:17:52.426 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster1-client.mirror.124947.log: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.124947.log.gz 2023-08-26T18:17:52.427 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.131313.log 2023-08-26T18:17:52.428 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster2-client.mirror.131313.log: 89.8% -- replaced with /var/log/ceph/cluster2-client.mirror.131313.log.gz 2023-08-26T18:17:52.428 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.119721.log 2023-08-26T18:17:52.429 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster2-client.mirror.119721.log: 89.7% -- replaced with /var/log/ceph/cluster2-client.mirror.119721.log.gz 2023-08-26T18:17:52.429 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.132572.log 2023-08-26T18:17:52.431 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster2-client.mirror.132572.log: 92.6% -- replaced with /var/log/ceph/cluster2-client.mirror.132572.log.gz 2023-08-26T18:17:52.431 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.135401.log 2023-08-26T18:17:52.432 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster2-client.mirror.135401.log: 87.9% -- replaced with /var/log/ceph/cluster2-client.mirror.135401.log.gz 2023-08-26T18:17:52.433 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.3.106251.log 2023-08-26T18:17:52.436 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster2-client.mirror.3.106251.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.110984.log 2023-08-26T18:17:52.444 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster1-client.mirror.110984.log: 93.4% -- replaced with /var/log/ceph/cluster1-client.mirror.110984.log.gz 2023-08-26T18:17:52.445 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.116436.log 2023-08-26T18:17:52.446 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster2-client.mirror.116436.log: 79.5% -- replaced with /var/log/ceph/cluster2-client.mirror.116436.log.gz 2023-08-26T18:17:52.446 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.116473.log 2023-08-26T18:17:52.447 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster2-client.mirror.116473.log: 87.5% -- replaced with /var/log/ceph/cluster2-client.mirror.116473.log.gz 2023-08-26T18:17:52.447 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.122623.log 2023-08-26T18:17:52.447 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster1-client.mirror.122623.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.122623.log.gz 2023-08-26T18:17:52.448 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.113815.log 2023-08-26T18:17:52.452 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster2-client.mirror.113815.log: 93.4% -- replaced with /var/log/ceph/cluster2-client.mirror.113815.log.gz 2023-08-26T18:17:52.452 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.141531.log 2023-08-26T18:17:52.453 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster1-client.mirror.141531.log: 86.8% -- replaced with /var/log/ceph/cluster1-client.mirror.141531.log.gz 2023-08-26T18:17:52.453 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.123959.log 2023-08-26T18:17:52.454 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster2-client.mirror.123959.log: 89.8% -- replaced with /var/log/ceph/cluster2-client.mirror.123959.log.gz 2023-08-26T18:17:52.454 INFO:teuthology.orchestra.run.smithi124.stderr: 93.8% -- replaced with /var/log/ceph/cluster2-client.mirror.3.106251.log.gz 2023-08-26T18:17:52.455 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.113186.log 2023-08-26T18:17:52.455 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster2-client.mirror.113186.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.133651.log 2023-08-26T18:17:52.456 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster2-client.mirror.133651.log: 88.0% -- replaced with /var/log/ceph/cluster2-client.mirror.133651.log.gz 2023-08-26T18:17:52.456 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.124267.log 2023-08-26T18:17:52.457 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster1-client.mirror.124267.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.124267.log.gz 2023-08-26T18:17:52.457 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.129143.log 2023-08-26T18:17:52.458 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster1-client.mirror.129143.log: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.129143.log.gz 2023-08-26T18:17:52.459 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.136251.log 2023-08-26T18:17:52.460 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster2-client.mirror.136251.log: 88.1% -- replaced with /var/log/ceph/cluster2-client.mirror.136251.log.gz 2023-08-26T18:17:52.460 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.127558.log 2023-08-26T18:17:52.461 INFO:teuthology.orchestra.run.smithi124.stderr: 93.4% -- replaced with /var/log/ceph/cluster2-client.mirror.113186.log.gz 2023-08-26T18:17:52.461 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.0.115070.log 2023-08-26T18:17:52.462 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster2-client.mirror.127558.log: 89.8% -- replaced with /var/log/ceph/cluster2-client.mirror.127558.log.gz 2023-08-26T18:17:52.462 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster2-client.mirror.0.115070.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.119583.log 2023-08-26T18:17:52.463 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster2-client.mirror.119583.log: 87.5% -- replaced with /var/log/ceph/cluster2-client.mirror.119583.log.gz 2023-08-26T18:17:52.463 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.104577.log 2023-08-26T18:17:52.475 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster2-client.admin.104577.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.104577.log.gz 2023-08-26T18:17:52.476 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.106677.log 2023-08-26T18:17:52.488 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster1-client.mirror.106677.log: 56.7% -- replaced with /var/log/ceph/cluster1-client.mirror.106677.log.gz 2023-08-26T18:17:52.488 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.140792.log 2023-08-26T18:17:52.499 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster2-client.mirror.140792.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.132313.log 2023-08-26T18:17:52.508 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster1-client.mirror.132313.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.124553.log 2023-08-26T18:17:52.508 INFO:teuthology.orchestra.run.smithi124.stderr: 87.6% -- replaced with /var/log/ceph/cluster1-client.mirror.132313.log.gz 2023-08-26T18:17:52.509 INFO:teuthology.orchestra.run.smithi124.stderr: 93.7% -- replaced with /var/log/ceph/cluster2-client.mirror.116300.log.gz 2023-08-26T18:17:52.509 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster1-client.mirror.124553.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.124553.log.gz 2023-08-26T18:17:52.510 INFO:teuthology.orchestra.run.smithi124.stderr: 93.9% -- replaced with /var/log/ceph/cluster2-client.mirror.0.115070.log.gz 2023-08-26T18:17:52.514 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.0.128374.log 2023-08-26T18:17:52.514 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.137893.log 2023-08-26T18:17:52.515 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster1-client.mirror.0.128374.log: /var/log/ceph/cluster1-client.mirror.137893.log: 86.8% -- replaced with /var/log/ceph/cluster1-client.mirror.137893.log.gz 2023-08-26T18:17:52.515 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.111796.log 2023-08-26T18:17:52.520 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster2-client.mirror.111796.log: 90.0% -- replaced with /var/log/ceph/cluster2-client.mirror.111796.log.gz 2023-08-26T18:17:52.521 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.120798.log 2023-08-26T18:17:52.522 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster2-client.mirror.120798.log: 89.8% -- replaced with /var/log/ceph/cluster2-client.mirror.120798.log.gz 2023-08-26T18:17:52.522 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.132376.log 2023-08-26T18:17:52.524 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster2-client.mirror.132376.log: 89.8% -- replaced with /var/log/ceph/cluster2-client.mirror.132376.log.gz 2023-08-26T18:17:52.524 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.122849.log 2023-08-26T18:17:52.525 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster1-client.mirror.122849.log: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.122849.log.gz 2023-08-26T18:17:52.525 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.145731.log 2023-08-26T18:17:52.526 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster1-client.admin.145731.log: 9.2% -- replaced with /var/log/ceph/cluster1-client.admin.145731.log.gz 2023-08-26T18:17:52.526 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.130928.log 2023-08-26T18:17:52.527 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster1-client.mirror.130928.log: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.130928.log.gz 2023-08-26T18:17:52.528 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.117596.log 2023-08-26T18:17:52.538 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster2-client.mirror.117596.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.109773.log 2023-08-26T18:17:52.542 INFO:teuthology.orchestra.run.smithi124.stderr: 93.5% -- replaced with /var/log/ceph/cluster1-client.mirror.0.128374.log.gz 2023-08-26T18:17:52.548 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster2-client.mirror.109773.log: 93.4% -- replaced with /var/log/ceph/cluster2-client.mirror.109773.log.gz 2023-08-26T18:17:52.549 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.133838.log 2023-08-26T18:17:52.551 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster2-client.mirror.133838.log: 93.7% -- replaced with /var/log/ceph/cluster2-client.mirror.115356.log.gz 2023-08-26T18:17:52.551 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.124075.log 2023-08-26T18:17:52.551 INFO:teuthology.orchestra.run.smithi124.stderr: 88.0% -- replaced with /var/log/ceph/cluster2-client.mirror.133838.log.gz 2023-08-26T18:17:52.552 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster1-client.mirror.124075.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.116146.log 2023-08-26T18:17:52.552 INFO:teuthology.orchestra.run.smithi124.stderr: 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.124075.log.gz 2023-08-26T18:17:52.553 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster2-client.mirror.116146.log: 79.6% -- replaced with /var/log/ceph/cluster2-client.mirror.116146.log.gz 2023-08-26T18:17:52.553 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.106050.log 2023-08-26T18:17:52.554 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster2-client.mirror.106050.log: 81.0% -- replaced with /var/log/ceph/cluster2-client.mirror.106050.log.gz 2023-08-26T18:17:52.554 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.123325.log 2023-08-26T18:17:52.554 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster1-client.mirror.123325.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.123325.log.gz 2023-08-26T18:17:52.555 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.123575.log 2023-08-26T18:17:52.555 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster1-client.mirror.123575.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.123575.log.gz 2023-08-26T18:17:52.556 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.119546.log 2023-08-26T18:17:52.557 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster2-client.mirror.119546.log: 79.6% -- replaced with /var/log/ceph/cluster2-client.mirror.119546.log.gz 2023-08-26T18:17:52.557 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.126262.log 2023-08-26T18:17:52.558 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster1-client.mirror.126262.log: 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.126262.log.gz 2023-08-26T18:17:52.558 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.132334.log 2023-08-26T18:17:52.559 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster2-client.mirror.132334.log: 92.3% -- replaced with /var/log/ceph/cluster2-client.mirror.132334.log.gz 2023-08-26T18:17:52.560 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.116205.log 2023-08-26T18:17:52.561 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.119494.log 2023-08-26T18:17:52.564 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster2-client.mirror.116205.log: /var/log/ceph/cluster2-client.mirror.119494.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.121389.log 2023-08-26T18:17:52.581 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster1-client.mirror.121389.log: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.121389.log.gz 2023-08-26T18:17:52.581 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.133962.log 2023-08-26T18:17:52.595 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster2-client.mirror.133962.log: 88.0% -- replaced with /var/log/ceph/cluster2-client.mirror.133962.log.gz 2023-08-26T18:17:52.596 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.136457.log 2023-08-26T18:17:52.637 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster2-client.mirror.136457.log: 90.4% -- replaced with /var/log/ceph/cluster2-client.mirror.136457.log.gz 2023-08-26T18:17:52.638 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.104499.log 2023-08-26T18:17:52.639 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster2-client.admin.104499.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.104499.log.gz 2023-08-26T18:17:52.639 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.111439.log 2023-08-26T18:17:52.643 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster2-client.mirror.111439.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.137465.log 2023-08-26T18:17:52.645 INFO:teuthology.orchestra.run.smithi124.stderr: 93.4% -- replaced with /var/log/ceph/cluster2-client.mirror.111439.log.gz 2023-08-26T18:17:52.645 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster1-client.mirror.137465.log: 86.7% -- replaced with /var/log/ceph/cluster1-client.mirror.137465.log.gz 2023-08-26T18:17:52.645 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.134086.log 2023-08-26T18:17:52.646 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster2-client.mirror.134086.log: 88.1% -- replaced with /var/log/ceph/cluster2-client.mirror.134086.log.gz 2023-08-26T18:17:52.646 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.137128.log 2023-08-26T18:17:52.647 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster1-client.mirror.137128.log: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.137128.log.gz 2023-08-26T18:17:52.647 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.134135.log 2023-08-26T18:17:52.649 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster2-client.mirror.134135.log: 90.5% -- replaced with /var/log/ceph/cluster2-client.mirror.134135.log.gz 2023-08-26T18:17:52.650 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.106643.log 2023-08-26T18:17:52.650 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster1-client.mirror.106643.log: 2.3% -- replaced with /var/log/ceph/cluster1-client.mirror.106643.log.gz 2023-08-26T18:17:52.651 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.114677.log 2023-08-26T18:17:52.651 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster2-client.mirror.114677.log: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.114677.log.gz 2023-08-26T18:17:52.652 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.108093.log 2023-08-26T18:17:52.652 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster1-client.mirror.108093.log: 73.6% -- replaced with /var/log/ceph/cluster1-client.mirror.108093.log.gz 2023-08-26T18:17:52.653 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.136814.log 2023-08-26T18:17:52.653 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster1-client.mirror.136814.log: 1.3% -- replaced with /var/log/ceph/cluster1-client.mirror.136814.log.gz 2023-08-26T18:17:52.654 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.128679.log 2023-08-26T18:17:52.654 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster1-client.mirror.128679.log: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.128679.log.gz 2023-08-26T18:17:52.654 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.137150.log 2023-08-26T18:17:52.655 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster1-client.mirror.137150.log: 86.8% -- replaced with /var/log/ceph/cluster1-client.mirror.137150.log.gz 2023-08-26T18:17:52.655 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.2.125946.log 2023-08-26T18:17:52.666 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster1-client.mirror.2.125946.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.111006.log 2023-08-26T18:17:52.681 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster2-client.mirror.111006.log: 89.9% -- replaced with /var/log/ceph/cluster2-client.mirror.111006.log.gz 2023-08-26T18:17:52.683 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.119704.log 2023-08-26T18:17:52.683 INFO:teuthology.orchestra.run.smithi124.stderr: 93.5% -- replaced with /var/log/ceph/cluster1-client.mirror.2.125946.log.gz 2023-08-26T18:17:52.684 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster2-client.mirror.119704.log: 79.5% -- replaced with /var/log/ceph/cluster2-client.mirror.119704.log.gz 2023-08-26T18:17:52.684 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.137555.log 2023-08-26T18:17:52.685 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster1-client.mirror.137555.log: 86.7% -- replaced with /var/log/ceph/cluster1-client.mirror.137555.log.gz 2023-08-26T18:17:52.685 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.131739.log 2023-08-26T18:17:52.685 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster1-client.mirror.131739.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.131739.log.gz 2023-08-26T18:17:52.685 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.1.111666.log 2023-08-26T18:17:52.694 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster2-client.mirror.1.111666.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.127207.log 2023-08-26T18:17:52.705 INFO:teuthology.orchestra.run.smithi124.stderr: 93.7% -- replaced with /var/log/ceph/cluster2-client.mirror.116923.log.gz 2023-08-26T18:17:52.705 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster2-client.mirror.127207.log: 92.4% -- replaced with /var/log/ceph/cluster2-client.mirror.127207.log.gz 2023-08-26T18:17:52.706 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.122757.log 2023-08-26T18:17:52.706 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster2-client.mirror.122757.log: 89.8% -- replaced with /var/log/ceph/cluster2-client.mirror.122757.log.gz 2023-08-26T18:17:52.706 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.113069.log 2023-08-26T18:17:52.707 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster1-client.mirror.113069.log: 89.3% -- replaced with /var/log/ceph/cluster1-client.mirror.113069.log.gz 2023-08-26T18:17:52.707 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.118647.log 2023-08-26T18:17:52.718 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster2-client.mirror.118647.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.145056.log 2023-08-26T18:17:52.729 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster2-client.admin.145056.log: 9.2% -- replaced with /var/log/ceph/cluster2-client.admin.145056.log.gz 2023-08-26T18:17:52.740 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.114417.log 2023-08-26T18:17:52.757 INFO:teuthology.orchestra.run.smithi124.stderr: 93.7% -- replaced with /var/log/ceph/cluster2-client.mirror.119494.log.gz 2023-08-26T18:17:52.759 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster2-client.mirror.114417.log: 88.5% -- replaced with /var/log/ceph/cluster2-client.mirror.114417.log.gz 2023-08-26T18:17:52.759 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.134926.log 2023-08-26T18:17:52.778 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster2-client.mirror.134926.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.135816.log 2023-08-26T18:17:52.789 INFO:teuthology.orchestra.run.smithi124.stderr: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.134926.log.gz 2023-08-26T18:17:52.796 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster2-client.mirror.135816.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.135422.log 2023-08-26T18:17:52.796 INFO:teuthology.orchestra.run.smithi124.stderr: 90.5% -- replaced with /var/log/ceph/cluster2-client.mirror.135816.log.gz 2023-08-26T18:17:52.798 INFO:teuthology.orchestra.run.smithi124.stderr: 93.8% -- replaced with /var/log/ceph/cluster2-client.mirror.1.111666.log.gz 2023-08-26T18:17:52.799 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster2-client.mirror.135422.log: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.135422.log.gz 2023-08-26T18:17:52.799 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.114864.log 2023-08-26T18:17:52.800 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster1-client.mirror.114864.log: 86.8% -- replaced with /var/log/ceph/cluster1-client.mirror.114864.log.gz 2023-08-26T18:17:52.800 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.142839.log 2023-08-26T18:17:52.801 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster1-client.admin.142839.log: 70.4% -- replaced with /var/log/ceph/cluster1-client.admin.142839.log.gz 2023-08-26T18:17:52.801 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.117208.log 2023-08-26T18:17:52.802 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster2-client.mirror.117208.log: 89.7% -- replaced with /var/log/ceph/cluster2-client.mirror.117208.log.gz 2023-08-26T18:17:52.802 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.108001.log 2023-08-26T18:17:52.803 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster1-client.mirror.108001.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.108001.log.gz 2023-08-26T18:17:52.804 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.124766.log 2023-08-26T18:17:52.805 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster1-client.mirror.124766.log: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.124766.log.gz 2023-08-26T18:17:52.805 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.135961.log 2023-08-26T18:17:52.805 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster2-client.mirror.135961.log: 88.0% -- replaced with /var/log/ceph/cluster2-client.mirror.135961.log.gz 2023-08-26T18:17:52.806 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.119842.log 2023-08-26T18:17:52.816 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster2-client.mirror.119842.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.121083.log 2023-08-26T18:17:52.839 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster2-client.mirror.121083.log: 89.8% -- replaced with /var/log/ceph/cluster2-client.mirror.121083.log.gz 2023-08-26T18:17:52.839 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.133480.log 2023-08-26T18:17:52.850 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster2-client.mirror.133480.log: 87.4% -- replaced with /var/log/ceph/cluster2-client.mirror.133480.log.gz 2023-08-26T18:17:52.850 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.107640.log 2023-08-26T18:17:52.861 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster2-client.mirror.107640.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.124191.log 2023-08-26T18:17:52.890 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster2-client.mirror.124191.log: 89.9% -- replaced with /var/log/ceph/cluster2-client.mirror.124191.log.gz 2023-08-26T18:17:52.891 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.108291.log 2023-08-26T18:17:52.912 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster2-client.mirror.108291.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.126642.log 2023-08-26T18:17:52.925 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster1-client.mirror.126642.log: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.126642.log.gz 2023-08-26T18:17:52.925 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.120485.log 2023-08-26T18:17:52.934 INFO:teuthology.orchestra.run.smithi124.stderr: 93.7% -- replaced with /var/log/ceph/cluster2-client.mirror.116205.log.gz 2023-08-26T18:17:52.939 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster1-client.mirror.120485.log: 93.7% -- replaced with /var/log/ceph/cluster2-client.mirror.117596.log.gz 2023-08-26T18:17:52.939 INFO:teuthology.orchestra.run.smithi124.stderr: 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.120485.log.gz 2023-08-26T18:17:52.940 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.1.111156.log 2023-08-26T18:17:52.950 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster1-client.mirror.1.111156.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.135713.log 2023-08-26T18:17:52.960 INFO:teuthology.orchestra.run.smithi124.stderr: 92.6% -- replaced with /var/log/ceph/cluster1-client.mirror.1.111156.log.gz 2023-08-26T18:17:52.965 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster2-client.mirror.135713.log: 87.9% -- replaced with /var/log/ceph/cluster2-client.mirror.135713.log.gz 2023-08-26T18:17:52.971 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.138140.log 2023-08-26T18:17:52.983 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster1-client.mirror.138140.log: 86.8% -- replaced with /var/log/ceph/cluster1-client.mirror.138140.log.gz 2023-08-26T18:17:52.983 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.121750.log 2023-08-26T18:17:52.995 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster2-client.mirror.121750.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.107760.log 2023-08-26T18:17:52.996 INFO:teuthology.orchestra.run.smithi124.stderr: 89.6% -- replaced with /var/log/ceph/cluster2-client.mirror.121750.log.gz 2023-08-26T18:17:53.007 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster2-client.mirror.107760.log: 81.7% -- replaced with /var/log/ceph/cluster2-client.mirror.107760.log.gz 2023-08-26T18:17:53.017 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.143096.log 2023-08-26T18:17:53.028 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster1-client.mirror.143096.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.143096.log.gz 2023-08-26T18:17:53.028 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.125675.log 2023-08-26T18:17:53.040 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster1-client.mirror.125675.log: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.125675.log.gz 2023-08-26T18:17:53.041 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.115811.log 2023-08-26T18:17:53.050 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster2-client.mirror.115811.log: 79.4% -- replaced with /var/log/ceph/cluster2-client.mirror.115811.log.gz 2023-08-26T18:17:53.051 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.122347.log 2023-08-26T18:17:53.057 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster1-client.mirror.122347.log: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.122347.log.gz 2023-08-26T18:17:53.057 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.143802.log 2023-08-26T18:17:53.072 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster2-client.mirror.143802.log: 8.2% -- replaced with /var/log/ceph/cluster2-client.mirror.143802.log.gz 2023-08-26T18:17:53.073 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.115722.log 2023-08-26T18:17:53.095 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster2-client.mirror.115722.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.143172.log 2023-08-26T18:17:53.108 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster1-client.admin.143172.log: 2.3% -- replaced with /var/log/ceph/cluster1-client.admin.143172.log.gz 2023-08-26T18:17:53.109 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.131832.log 2023-08-26T18:17:53.110 INFO:teuthology.orchestra.run.smithi124.stderr: 94.0% -- replaced with /var/log/ceph/cluster2-client.mirror.115722.log.gz 2023-08-26T18:17:53.110 INFO:teuthology.orchestra.run.smithi124.stderr: 93.7% -- replaced with /var/log/ceph/cluster2-client.mirror.140792.log.gz 2023-08-26T18:17:53.112 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster2-client.mirror.131832.log: 92.5% -- replaced with /var/log/ceph/cluster2-client.mirror.131832.log.gz 2023-08-26T18:17:53.112 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.132551.log 2023-08-26T18:17:53.113 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster1-client.mirror.132551.log: 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.132551.log.gz 2023-08-26T18:17:53.113 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.136519.log 2023-08-26T18:17:53.116 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster2-client.mirror.136519.log: 90.4% -- replaced with /var/log/ceph/cluster2-client.mirror.136519.log.gz 2023-08-26T18:17:53.116 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.120219.log 2023-08-26T18:17:53.117 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster2-client.mirror.120219.log: 1.3% -- replaced with /var/log/ceph/cluster2-client.mirror.120219.log.gz 2023-08-26T18:17:53.117 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.132771.log 2023-08-26T18:17:53.118 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster1-client.mirror.132771.log: 87.5% -- replaced with /var/log/ceph/cluster1-client.mirror.132771.log.gz 2023-08-26T18:17:53.118 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.119267.log 2023-08-26T18:17:53.119 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster2-client.mirror.119267.log: 79.7% -- replaced with /var/log/ceph/cluster2-client.mirror.119267.log.gz 2023-08-26T18:17:53.120 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.108712.log 2023-08-26T18:17:53.120 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster1-client.admin.108712.log: 87.6% -- replaced with /var/log/ceph/cluster1-client.admin.108712.log.gz 2023-08-26T18:17:53.120 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.129492.log 2023-08-26T18:17:53.123 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster2-client.mirror.129492.log: 92.5% -- replaced with /var/log/ceph/cluster2-client.mirror.129492.log.gz 2023-08-26T18:17:53.123 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.109234.log 2023-08-26T18:17:53.124 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster1-client.mirror.109234.log: 87.9% -- replaced with /var/log/ceph/cluster1-client.mirror.109234.log.gz 2023-08-26T18:17:53.124 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.138230.log 2023-08-26T18:17:53.125 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster1-client.mirror.138230.log: 86.8% -- replaced with /var/log/ceph/cluster1-client.mirror.138230.log.gz 2023-08-26T18:17:53.125 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.113491.log 2023-08-26T18:17:53.131 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster1-client.mirror.113491.log: 93.4% -- replaced with /var/log/ceph/cluster1-client.mirror.113491.log.gz 2023-08-26T18:17:53.131 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.114906.log 2023-08-26T18:17:53.132 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster2-client.mirror.114906.log: 89.7% -- replaced with /var/log/ceph/cluster2-client.mirror.114906.log.gz 2023-08-26T18:17:53.134 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.135734.log 2023-08-26T18:17:53.134 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster2-client.mirror.135734.log: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.135734.log.gz 2023-08-26T18:17:53.135 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.114073.log 2023-08-26T18:17:53.142 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster2-client.mirror.114073.log: 93.4% -- replaced with /var/log/ceph/cluster2-client.mirror.114073.log.gz 2023-08-26T18:17:53.142 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.106833.log 2023-08-26T18:17:53.143 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster2-client.admin.106833.log: 66.4% -- replaced with /var/log/ceph/cluster2-client.admin.106833.log.gz 2023-08-26T18:17:53.144 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.136271.log 2023-08-26T18:17:53.147 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster2-client.mirror.136271.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.134449.log 2023-08-26T18:17:53.148 INFO:teuthology.orchestra.run.smithi124.stderr: 90.4% -- replaced with /var/log/ceph/cluster2-client.mirror.136271.log.gz 2023-08-26T18:17:53.151 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster2-client.mirror.134449.log: 90.4% -- replaced with /var/log/ceph/cluster2-client.mirror.134449.log.gz 2023-08-26T18:17:53.152 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.126138.log 2023-08-26T18:17:53.153 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster2-client.mirror.126138.log: 89.8% -- replaced with /var/log/ceph/cluster2-client.mirror.126138.log.gz 2023-08-26T18:17:53.153 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.113406.log 2023-08-26T18:17:53.154 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster1-client.mirror.113406.log: 88.5% -- replaced with /var/log/ceph/cluster1-client.mirror.113406.log.gz 2023-08-26T18:17:53.155 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.132527.log 2023-08-26T18:17:53.155 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster1-client.mirror.132527.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.132527.log.gz 2023-08-26T18:17:53.157 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.138275.log 2023-08-26T18:17:53.157 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster1-client.mirror.138275.log: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.138275.log.gz 2023-08-26T18:17:53.158 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.120526.log 2023-08-26T18:17:53.159 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster1-client.mirror.120526.log: 92.1% -- replaced with /var/log/ceph/cluster1-client.mirror.120526.log.gz 2023-08-26T18:17:53.159 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.136858.log 2023-08-26T18:17:53.160 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster1-client.mirror.136858.log: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.136858.log.gz 2023-08-26T18:17:53.160 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.106867.log 2023-08-26T18:17:53.161 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster2-client.mirror.106867.log: 2.3% -- replaced with /var/log/ceph/cluster2-client.mirror.106867.log.gz 2023-08-26T18:17:53.161 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.139018.log 2023-08-26T18:17:53.162 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster1-client.mirror.139018.log: 86.7% -- replaced with /var/log/ceph/cluster1-client.mirror.139018.log.gz 2023-08-26T18:17:53.162 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.105928.log 2023-08-26T18:17:53.163 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster2-client.admin.105928.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.105928.log.gz 2023-08-26T18:17:53.163 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.126214.log 2023-08-26T18:17:53.164 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster1-client.mirror.126214.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.126214.log.gz 2023-08-26T18:17:53.164 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.121687.log 2023-08-26T18:17:53.165 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster1-client.mirror.121687.log: 87.6% -- replaced with /var/log/ceph/cluster1-client.mirror.121687.log.gz 2023-08-26T18:17:53.165 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.124138.log 2023-08-26T18:17:53.166 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster2-client.mirror.124138.log: 89.8% -- replaced with /var/log/ceph/cluster2-client.mirror.124138.log.gz 2023-08-26T18:17:53.167 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.105882.log 2023-08-26T18:17:53.167 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster2-client.admin.105882.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.105882.log.gz 2023-08-26T18:17:53.168 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.114202.log 2023-08-26T18:17:53.173 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster2-client.mirror.114202.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.120506.log 2023-08-26T18:17:53.174 INFO:teuthology.orchestra.run.smithi124.stderr: 93.4% -- replaced with /var/log/ceph/cluster2-client.mirror.114202.log.gz 2023-08-26T18:17:53.179 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster2-client.mirror.120506.log: 92.3% -- replaced with /var/log/ceph/cluster2-client.mirror.120506.log.gz 2023-08-26T18:17:53.180 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.127602.log 2023-08-26T18:17:53.180 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster1-client.mirror.127602.log: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.127602.log.gz 2023-08-26T18:17:53.180 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.127340.log 2023-08-26T18:17:53.181 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster1-client.mirror.127340.log: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.127340.log.gz 2023-08-26T18:17:53.181 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.140879.log 2023-08-26T18:17:53.181 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster1-client.mirror.140879.log: 86.8% -- replaced with /var/log/ceph/cluster1-client.mirror.140879.log.gz 2023-08-26T18:17:53.182 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.105514.log 2023-08-26T18:17:53.182 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster1-client.admin.105514.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.105514.log.gz 2023-08-26T18:17:53.183 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.110334.log 2023-08-26T18:17:53.183 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster2-client.mirror.110334.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.124378.log 2023-08-26T18:17:53.186 INFO:teuthology.orchestra.run.smithi124.stderr: 93.7% -- replaced with /var/log/ceph/cluster2-client.mirror.118647.log.gz 2023-08-26T18:17:53.187 INFO:teuthology.orchestra.run.smithi124.stderr: 93.3% -- replaced with /var/log/ceph/cluster2-client.mirror.110334.log.gz 2023-08-26T18:17:53.187 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.143415.log 2023-08-26T18:17:53.188 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster2-client.mirror.124378.log: 89.7% -- replaced with /var/log/ceph/cluster2-client.mirror.124378.log.gz 2023-08-26T18:17:53.189 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.140964.log 2023-08-26T18:17:53.189 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster2-client.admin.143415.log: 8.2% -- replaced with /var/log/ceph/cluster2-client.admin.143415.log.gz 2023-08-26T18:17:53.192 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster2-client.mirror.140964.log: 91.6% -- replaced with /var/log/ceph/cluster2-client.mirror.140964.log.gz 2023-08-26T18:17:53.192 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.112298.log 2023-08-26T18:17:53.192 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.111959.log 2023-08-26T18:17:53.193 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster1-client.mirror.112298.log: 89.1% -- replaced with /var/log/ceph/cluster1-client.mirror.112298.log.gz 2023-08-26T18:17:53.193 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster2-client.mirror.111959.log: 93.8% -- replaced with /var/log/ceph/cluster2-client.mirror.108291.log.gz 2023-08-26T18:17:53.197 INFO:teuthology.orchestra.run.smithi124.stderr: 93.4% -- replaced with /var/log/ceph/cluster2-client.mirror.111959.log.gz 2023-08-26T18:17:53.198 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.128247.log 2023-08-26T18:17:53.199 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.142050.log 2023-08-26T18:17:53.199 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster1-client.mirror.128247.log: /var/log/ceph/cluster2-client.mirror.142050.log: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.128247.log.gz 2023-08-26T18:17:53.199 INFO:teuthology.orchestra.run.smithi124.stderr: 88.1% -- replaced with /var/log/ceph/cluster2-client.mirror.142050.log.gz 2023-08-26T18:17:53.200 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.141443.log 2023-08-26T18:17:53.200 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.109968.log 2023-08-26T18:17:53.201 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster1-client.mirror.141443.log: /var/log/ceph/cluster2-client.mirror.109968.log: 87.2% -- replaced with /var/log/ceph/cluster1-client.mirror.141443.log.gz 2023-08-26T18:17:53.202 INFO:teuthology.orchestra.run.smithi124.stderr: 91.8% -- replaced with /var/log/ceph/cluster2-client.mirror.109968.log.gz 2023-08-26T18:17:53.202 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.106974.log 2023-08-26T18:17:53.203 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.110008.log 2023-08-26T18:17:53.203 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster2-client.mirror.106974.log: 79.7% -- replaced with /var/log/ceph/cluster2-client.mirror.106974.log.gz 2023-08-26T18:17:53.203 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster1-client.mirror.110008.log: 73.0% -- replaced with /var/log/ceph/cluster1-client.mirror.110008.log.gz 2023-08-26T18:17:53.203 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.140532.log 2023-08-26T18:17:53.204 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.145621.log 2023-08-26T18:17:53.205 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster2-client.mirror.140532.log: /var/log/ceph/cluster1-client.admin.145621.log: 8.2% -- replaced with /var/log/ceph/cluster1-client.admin.145621.log.gz 2023-08-26T18:17:53.205 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.110964.log 2023-08-26T18:17:53.209 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster2-client.mirror.110964.log: 93.4% -- replaced with /var/log/ceph/cluster2-client.mirror.110964.log.gz 2023-08-26T18:17:53.209 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.122006.log 2023-08-26T18:17:53.210 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster2-client.mirror.122006.log: 89.8% -- replaced with /var/log/ceph/cluster2-client.mirror.122006.log.gz 2023-08-26T18:17:53.211 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.132084.log 2023-08-26T18:17:53.213 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster1-client.mirror.132084.log: 92.3% -- replaced with /var/log/ceph/cluster1-client.mirror.132084.log.gz 2023-08-26T18:17:53.213 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.115906.log 2023-08-26T18:17:53.214 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster2-client.mirror.115906.log: 79.6% -- replaced with /var/log/ceph/cluster2-client.mirror.115906.log.gz 2023-08-26T18:17:53.214 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.120548.log 2023-08-26T18:17:53.216 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster2-client.mirror.120548.log: 89.6% -- replaced with /var/log/ceph/cluster2-client.mirror.120548.log.gz 2023-08-26T18:17:53.216 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.141552.log 2023-08-26T18:17:53.217 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster2-client.mirror.141552.log: 88.1% -- replaced with /var/log/ceph/cluster2-client.mirror.141552.log.gz 2023-08-26T18:17:53.218 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.117799.log 2023-08-26T18:17:53.219 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster2-client.mirror.117799.log: 89.7% -- replaced with /var/log/ceph/cluster2-client.mirror.117799.log.gz 2023-08-26T18:17:53.219 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.122226.log 2023-08-26T18:17:53.221 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster2-client.mirror.122226.log: 89.7% -- replaced with /var/log/ceph/cluster2-client.mirror.122226.log.gz 2023-08-26T18:17:53.221 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.125989.log 2023-08-26T18:17:53.223 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster2-client.mirror.125989.log: 93.7% -- replaced with /var/log/ceph/cluster2-client.mirror.119842.log.gz 2023-08-26T18:17:53.223 INFO:teuthology.orchestra.run.smithi124.stderr: 92.4% -- replaced with /var/log/ceph/cluster2-client.mirror.125989.log.gz 2023-08-26T18:17:53.224 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.116811.log 2023-08-26T18:17:53.225 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.111866.log 2023-08-26T18:17:53.225 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster2-client.mirror.116811.log: /var/log/ceph/cluster1-client.mirror.111866.log: 88.9% -- replaced with /var/log/ceph/cluster1-client.mirror.111866.log.gz 2023-08-26T18:17:53.225 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.126190.log 2023-08-26T18:17:53.226 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster1-client.mirror.126190.log: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.126190.log.gz 2023-08-26T18:17:53.226 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.112831.log 2023-08-26T18:17:53.227 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster1-client.mirror.112831.log: 87.9% -- replaced with /var/log/ceph/cluster1-client.mirror.112831.log.gz 2023-08-26T18:17:53.228 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.118509.log 2023-08-26T18:17:53.228 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster2-client.mirror.118509.log: 79.3% -- replaced with /var/log/ceph/cluster2-client.mirror.118509.log.gz 2023-08-26T18:17:53.228 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.119067.log 2023-08-26T18:17:53.229 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster2-client.mirror.119067.log: 87.4% -- replaced with /var/log/ceph/cluster2-client.mirror.119067.log.gz 2023-08-26T18:17:53.229 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.145160.log 2023-08-26T18:17:53.230 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster2-client.mirror.145160.log: 7.1% -- replaced with /var/log/ceph/cluster2-client.mirror.145160.log.gz 2023-08-26T18:17:53.230 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.115030.log 2023-08-26T18:17:53.231 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster2-client.mirror.115030.log: 89.7% -- replaced with /var/log/ceph/cluster2-client.mirror.115030.log.gz 2023-08-26T18:17:53.231 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.108206.log 2023-08-26T18:17:53.232 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster1-client.mirror.108206.log: 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.108206.log.gz 2023-08-26T18:17:53.232 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.107717.log 2023-08-26T18:17:53.233 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster1-client.admin.107717.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.admin.107717.log.gz 2023-08-26T18:17:53.234 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.113165.log 2023-08-26T18:17:53.234 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster1-client.mirror.113165.log: 88.4% -- replaced with /var/log/ceph/cluster1-client.mirror.113165.log.gz 2023-08-26T18:17:53.235 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.110311.log 2023-08-26T18:17:53.241 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster1-client.mirror.110311.log: 93.4% -- replaced with /var/log/ceph/cluster1-client.mirror.110311.log.gz 2023-08-26T18:17:53.242 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.131250.log 2023-08-26T18:17:53.242 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster1-client.mirror.131250.log: 87.6% -- replaced with /var/log/ceph/cluster1-client.mirror.131250.log.gz 2023-08-26T18:17:53.243 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.143318.log 2023-08-26T18:17:53.243 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster1-client.mirror.143318.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.143318.log.gz 2023-08-26T18:17:53.244 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.119109.log 2023-08-26T18:17:53.244 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster2-client.mirror.119109.log: 79.7% -- replaced with /var/log/ceph/cluster2-client.mirror.119109.log.gz 2023-08-26T18:17:53.245 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.141312.log 2023-08-26T18:17:53.245 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster2-client.mirror.141312.log: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.141312.log.gz 2023-08-26T18:17:53.246 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.136333.log 2023-08-26T18:17:53.248 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster2-client.mirror.136333.log: 90.4% -- replaced with /var/log/ceph/cluster2-client.mirror.136333.log.gz 2023-08-26T18:17:53.248 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.141332.log 2023-08-26T18:17:53.251 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster2-client.mirror.141332.log: 91.6% -- replaced with /var/log/ceph/cluster2-client.mirror.141332.log.gz 2023-08-26T18:17:53.252 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.138073.log 2023-08-26T18:17:53.252 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster1-client.mirror.138073.log: 86.8% -- replaced with /var/log/ceph/cluster1-client.mirror.138073.log.gz 2023-08-26T18:17:53.253 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.138973.log 2023-08-26T18:17:53.263 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster1-client.mirror.138973.log: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.138973.log.gz 2023-08-26T18:17:53.264 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.142150.log 2023-08-26T18:17:53.264 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster2-client.mirror.142150.log: 92.9% -- replaced with /var/log/ceph/cluster2-client.mirror.142150.log.gz 2023-08-26T18:17:53.264 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.112442.log 2023-08-26T18:17:53.265 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster2-client.mirror.112442.log: 93.4% -- replaced with /var/log/ceph/cluster2-client.mirror.112442.log.gz 2023-08-26T18:17:53.265 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.118341.log 2023-08-26T18:17:53.266 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster1-client.mirror.118341.log: 95.3% -- replaced with /var/log/ceph/cluster1-client.mirror.118341.log.gz 2023-08-26T18:17:53.266 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.135837.log 2023-08-26T18:17:53.267 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster2-client.mirror.135837.log: 88.0% -- replaced with /var/log/ceph/cluster2-client.mirror.135837.log.gz 2023-08-26T18:17:53.267 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.131271.log 2023-08-26T18:17:53.271 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster2-client.mirror.131271.log: 92.4% -- replaced with /var/log/ceph/cluster2-client.mirror.131271.log.gz 2023-08-26T18:17:53.271 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.138433.log 2023-08-26T18:17:53.271 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster1-client.mirror.138433.log: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.138433.log.gz 2023-08-26T18:17:53.272 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.108480.log 2023-08-26T18:17:53.273 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster1-client.mirror.108480.log: 87.6% -- replaced with /var/log/ceph/cluster1-client.mirror.108480.log.gz 2023-08-26T18:17:53.273 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.140642.log 2023-08-26T18:17:53.274 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster1-client.mirror.140642.log: 86.8% -- replaced with /var/log/ceph/cluster1-client.mirror.140642.log.gz 2023-08-26T18:17:53.274 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.140944.log 2023-08-26T18:17:53.274 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster2-client.mirror.140944.log: 88.1% -- replaced with /var/log/ceph/cluster2-client.mirror.140944.log.gz 2023-08-26T18:17:53.274 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.126238.log 2023-08-26T18:17:53.276 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster1-client.mirror.126238.log: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.126238.log.gz 2023-08-26T18:17:53.276 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.134698.log 2023-08-26T18:17:53.277 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster2-client.mirror.134698.log: 90.4% -- replaced with /var/log/ceph/cluster2-client.mirror.134698.log.gz 2023-08-26T18:17:53.277 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.116067.log 2023-08-26T18:17:53.278 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster2-client.mirror.116067.log: 79.5% -- replaced with /var/log/ceph/cluster2-client.mirror.116067.log.gz 2023-08-26T18:17:53.278 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.135610.log 2023-08-26T18:17:53.279 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster2-client.mirror.135610.log: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.135610.log.gz 2023-08-26T18:17:53.281 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.120988.log 2023-08-26T18:17:53.282 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster2-client.mirror.120988.log: 92.4% -- replaced with /var/log/ceph/cluster2-client.mirror.120988.log.gz 2023-08-26T18:17:53.282 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.117060.log 2023-08-26T18:17:53.283 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster2-client.mirror.117060.log: 79.4% -- replaced with /var/log/ceph/cluster2-client.mirror.117060.log.gz 2023-08-26T18:17:53.283 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.127578.log 2023-08-26T18:17:53.283 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.113093.log 2023-08-26T18:17:53.284 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster2-client.mirror.127578.log: 87.3% -- replaced with /var/log/ceph/cluster2-client.mirror.127578.log.gz 2023-08-26T18:17:53.285 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster1-client.mirror.113093.log: 89.3% -- replaced with /var/log/ceph/cluster1-client.mirror.113093.log.gz 2023-08-26T18:17:53.285 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.128893.log 2023-08-26T18:17:53.285 INFO:teuthology.orchestra.run.smithi124.stderr:gzip /var/log/ceph/cluster2-client.mirror.128893.log: -5 --verbose -- /var/log/ceph/cluster1-client.mirror.123277.log 2023-08-26T18:17:53.285 INFO:teuthology.orchestra.run.smithi124.stderr: 87.3% -- replaced with /var/log/ceph/cluster2-client.mirror.128893.log.gz 2023-08-26T18:17:53.289 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster1-client.mirror.123277.log: 87.2% -- replaced with /var/log/ceph/cluster1-client.mirror.123277.log.gz 2023-08-26T18:17:53.289 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.133121.log 2023-08-26T18:17:53.290 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster2-client.mirror.133121.log: 87.4% -- replaced with /var/log/ceph/cluster2-client.mirror.133121.log.gz 2023-08-26T18:17:53.291 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.108227.log 2023-08-26T18:17:53.295 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster2-client.mirror.108227.log: 93.4% -- replaced with /var/log/ceph/cluster2-client.mirror.108227.log.gz 2023-08-26T18:17:53.296 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.145106.log 2023-08-26T18:17:53.296 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.123527.log 2023-08-26T18:17:53.297 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.130323.log 2023-08-26T18:17:53.297 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster1-client.mirror.123527.log: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.123527.log.gz 2023-08-26T18:17:53.297 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster1-client.mirror.130323.log: /var/log/ceph/cluster2-client.mirror.145106.log: 9.2% -- replaced with /var/log/ceph/cluster2-client.mirror.145106.log.gz 2023-08-26T18:17:53.298 INFO:teuthology.orchestra.run.smithi124.stderr: 92.2% -- replaced with /var/log/ceph/cluster1-client.mirror.130323.log.gz 2023-08-26T18:17:53.299 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.108573.log 2023-08-26T18:17:53.304 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster2-client.mirror.108573.log: 93.4% -- replaced with /var/log/ceph/cluster2-client.mirror.108573.log.gz 2023-08-26T18:17:53.304 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.138365.log 2023-08-26T18:17:53.306 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster1-client.mirror.138365.log: 86.8% -- replaced with /var/log/ceph/cluster1-client.mirror.138365.log.gz 2023-08-26T18:17:53.306 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.106884.log 2023-08-26T18:17:53.306 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster2-client.mirror.106884.log: 3.5% -- replaced with /var/log/ceph/cluster2-client.mirror.106884.log.gz 2023-08-26T18:17:53.306 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.120871.log 2023-08-26T18:17:53.307 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster2-client.mirror.120871.log: 87.3% -- replaced with /var/log/ceph/cluster2-client.mirror.120871.log.gz 2023-08-26T18:17:53.308 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.124734.log 2023-08-26T18:17:53.308 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster1-client.mirror.124734.log: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.124734.log.gz 2023-08-26T18:17:53.308 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.136561.log 2023-08-26T18:17:53.309 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster2-client.mirror.136561.log: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.136561.log.gz 2023-08-26T18:17:53.310 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.143027.log 2023-08-26T18:17:53.310 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster1-client.mirror.143027.log: 10.2% -- replaced with /var/log/ceph/cluster1-client.mirror.143027.log.gz 2023-08-26T18:17:53.310 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.114007.log 2023-08-26T18:17:53.315 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster1-client.mirror.114007.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.132699.log 2023-08-26T18:17:53.315 INFO:teuthology.orchestra.run.smithi124.stderr: 93.3% -- replaced with /var/log/ceph/cluster1-client.mirror.114007.log.gz 2023-08-26T18:17:53.316 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.116710.log 2023-08-26T18:17:53.317 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster1-client.mirror.132699.log: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.132699.log.gz 2023-08-26T18:17:53.317 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster2-client.mirror.116710.log: 87.5% -- replaced with /var/log/ceph/cluster2-client.mirror.116710.log.gz 2023-08-26T18:17:53.317 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.128772.log 2023-08-26T18:17:53.318 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster2-client.mirror.128772.log: 92.5% -- replaced with /var/log/ceph/cluster2-client.mirror.128772.log.gz 2023-08-26T18:17:53.319 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.129256.log 2023-08-26T18:17:53.320 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster1-client.mirror.129256.log: 92.1% -- replaced with /var/log/ceph/cluster1-client.mirror.129256.log.gz 2023-08-26T18:17:53.321 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.114799.log 2023-08-26T18:17:53.322 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster2-client.mirror.114799.log: 88.1% -- replaced with /var/log/ceph/cluster2-client.mirror.114799.log.gz 2023-08-26T18:17:53.322 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.134905.log 2023-08-26T18:17:53.323 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.125371.log 2023-08-26T18:17:53.325 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster2-client.mirror.134905.log: 88.0% -- replaced with /var/log/ceph/cluster2-client.mirror.134905.log.gz 2023-08-26T18:17:53.325 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster1-client.mirror.125371.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.125371.log.gz 2023-08-26T18:17:53.325 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.123855.log 2023-08-26T18:17:53.326 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.132919.log 2023-08-26T18:17:53.326 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster1-client.mirror.123855.log: 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.123855.log.gz 2023-08-26T18:17:53.326 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster1-client.mirror.132919.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.132919.log.gz 2023-08-26T18:17:53.326 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.134635.log 2023-08-26T18:17:53.327 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster2-client.mirror.134635.log: 90.4% -- replaced with /var/log/ceph/cluster2-client.mirror.134635.log.gz 2023-08-26T18:17:53.328 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.137735.log 2023-08-26T18:17:53.328 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster1-client.mirror.137735.log: 86.6% -- replaced with /var/log/ceph/cluster1-client.mirror.137735.log.gz 2023-08-26T18:17:53.329 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.119563.log 2023-08-26T18:17:53.330 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster2-client.mirror.119563.log: 89.7% -- replaced with /var/log/ceph/cluster2-client.mirror.119563.log.gz 2023-08-26T18:17:53.330 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.118489.log 2023-08-26T18:17:53.341 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster2-client.mirror.118489.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.125740.log 2023-08-26T18:17:53.343 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster1-client.mirror.125740.log: 92.2% -- replaced with /var/log/ceph/cluster1-client.mirror.125740.log.gz 2023-08-26T18:17:53.343 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.125651.log 2023-08-26T18:17:53.344 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster1-client.mirror.125651.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.125651.log.gz 2023-08-26T18:17:53.345 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.123620.log 2023-08-26T18:17:53.353 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster2-client.mirror.123620.log: 92.4% -- replaced with /var/log/ceph/cluster2-client.mirror.123620.log.gz 2023-08-26T18:17:53.354 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.136925.log 2023-08-26T18:17:53.354 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster1-client.mirror.136925.log: 86.8% -- replaced with /var/log/ceph/cluster1-client.mirror.136925.log.gz 2023-08-26T18:17:53.354 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.142874.log 2023-08-26T18:17:53.355 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster1-client.mirror.142874.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.142874.log.gz 2023-08-26T18:17:53.355 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.116732.log 2023-08-26T18:17:53.369 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster2-client.mirror.116732.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.105376.log 2023-08-26T18:17:53.378 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster2-client.admin.105376.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.105376.log.gz 2023-08-26T18:17:53.378 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.137780.log 2023-08-26T18:17:53.387 INFO:teuthology.orchestra.run.smithi124.stderr: 93.8% -- replaced with /var/log/ceph/cluster2-client.mirror.107640.log.gz 2023-08-26T18:17:53.391 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster1-client.mirror.137780.log: 86.8% -- replaced with /var/log/ceph/cluster1-client.mirror.137780.log.gz 2023-08-26T18:17:53.391 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.138455.log 2023-08-26T18:17:53.393 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster1-client.mirror.138455.log: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.138455.log.gz 2023-08-26T18:17:53.393 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.113206.log 2023-08-26T18:17:53.399 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster1-client.mirror.113206.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.127249.log 2023-08-26T18:17:53.400 INFO:teuthology.orchestra.run.smithi124.stderr: 93.4% -- replaced with /var/log/ceph/cluster1-client.mirror.113206.log.gz 2023-08-26T18:17:53.401 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster2-client.mirror.127249.log: 89.7% -- replaced with /var/log/ceph/cluster2-client.mirror.127249.log.gz 2023-08-26T18:17:53.401 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.124243.log 2023-08-26T18:17:53.401 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster1-client.mirror.124243.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.124243.log.gz 2023-08-26T18:17:53.402 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.143079.log 2023-08-26T18:17:53.402 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster1-client.mirror.143079.log: 10.2% -- replaced with /var/log/ceph/cluster1-client.mirror.143079.log.gz 2023-08-26T18:17:53.403 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.141353.log 2023-08-26T18:17:53.403 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster2-client.mirror.141353.log: 79.4% -- replaced with /var/log/ceph/cluster2-client.mirror.141353.log.gz 2023-08-26T18:17:53.404 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.128751.log 2023-08-26T18:17:53.405 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster1-client.mirror.128751.log: 87.6% -- replaced with /var/log/ceph/cluster1-client.mirror.128751.log.gz 2023-08-26T18:17:53.405 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.110651.log 2023-08-26T18:17:53.406 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster2-client.mirror.110651.log: 90.0% -- replaced with /var/log/ceph/cluster2-client.mirror.110651.log.gz 2023-08-26T18:17:53.407 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.129010.log 2023-08-26T18:17:53.409 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster2-client.mirror.129010.log: 92.4% -- replaced with /var/log/ceph/cluster2-client.mirror.129010.log.gz 2023-08-26T18:17:53.409 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.138725.log 2023-08-26T18:17:53.410 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster1-client.mirror.138725.log: 86.7% -- replaced with /var/log/ceph/cluster1-client.mirror.138725.log.gz 2023-08-26T18:17:53.410 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.106918.log 2023-08-26T18:17:53.411 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster2-client.mirror.106918.log: 57.4% -- replaced with /var/log/ceph/cluster2-client.mirror.106918.log.gz 2023-08-26T18:17:53.411 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.133921.log 2023-08-26T18:17:53.412 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster2-client.mirror.133921.log: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.133921.log.gz 2023-08-26T18:17:53.412 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.113141.log 2023-08-26T18:17:53.414 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster1-client.mirror.113141.log: 89.3% -- replaced with /var/log/ceph/cluster1-client.mirror.113141.log.gz 2023-08-26T18:17:53.415 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.124336.log 2023-08-26T18:17:53.415 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster2-client.mirror.124336.log: 92.5% -- replaced with /var/log/ceph/cluster2-client.mirror.124336.log.gz 2023-08-26T18:17:53.416 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.142746.log 2023-08-26T18:17:53.416 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster1-client.mirror.142746.log: 66.3% -- replaced with /var/log/ceph/cluster1-client.mirror.142746.log.gz 2023-08-26T18:17:53.417 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.109370.log 2023-08-26T18:17:53.420 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster2-client.mirror.109370.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.141750.log 2023-08-26T18:17:53.432 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster1-client.mirror.141750.log: 79.4% -- replaced with /var/log/ceph/cluster1-client.mirror.141750.log.gz 2023-08-26T18:17:53.432 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.137105.log 2023-08-26T18:17:53.444 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster1-client.mirror.137105.log: 86.8% -- replaced with /var/log/ceph/cluster1-client.mirror.137105.log.gz 2023-08-26T18:17:53.454 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.109326.log 2023-08-26T18:17:53.478 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster1-client.mirror.109326.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.141247.log 2023-08-26T18:17:53.478 INFO:teuthology.orchestra.run.smithi124.stderr: 93.4% -- replaced with /var/log/ceph/cluster1-client.mirror.109326.log.gz 2023-08-26T18:17:53.489 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster1-client.mirror.141247.log: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.141247.log.gz 2023-08-26T18:17:53.489 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.141487.log 2023-08-26T18:17:53.496 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.138118.log 2023-08-26T18:17:53.497 INFO:teuthology.orchestra.run.smithi124.stderr: 93.7% -- replaced with /var/log/ceph/cluster2-client.mirror.116811.log.gz 2023-08-26T18:17:53.498 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster1-client.mirror.138118.log: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.138118.log.gz 2023-08-26T18:17:53.498 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.144321.log 2023-08-26T18:17:53.499 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster2-client.mirror.144321.log: 7.1% -- replaced with /var/log/ceph/cluster2-client.mirror.144321.log.gz 2023-08-26T18:17:53.499 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.133262.log 2023-08-26T18:17:53.500 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster2-client.mirror.141487.log: 88.6% -- replaced with /var/log/ceph/cluster2-client.mirror.141487.log.gz 2023-08-26T18:17:53.503 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster2-client.mirror.133262.log: 92.5% -- replaced with /var/log/ceph/cluster2-client.mirror.133262.log.gz 2023-08-26T18:17:53.504 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.118301.log 2023-08-26T18:17:53.513 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster2-client.mirror.118301.log: 93.7% -- replaced with /var/log/ceph/cluster2-client.mirror.116732.log.gz 2023-08-26T18:17:53.513 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.0.114286.log 2023-08-26T18:17:53.524 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.114637.log 2023-08-26T18:17:53.524 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster1-client.mirror.0.114286.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.117740.log 2023-08-26T18:17:53.528 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.136313.log 2023-08-26T18:17:53.529 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.136623.log 2023-08-26T18:17:53.529 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.141985.log 2023-08-26T18:17:53.530 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.115870.log 2023-08-26T18:17:53.532 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.114926.log 2023-08-26T18:17:53.533 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.126488.log 2023-08-26T18:17:53.533 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.106082.log 2023-08-26T18:17:53.533 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.136540.log 2023-08-26T18:17:53.534 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.131787.log 2023-08-26T18:17:53.535 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.134678.log 2023-08-26T18:17:53.535 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.114717.log 2023-08-26T18:17:53.536 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.116183.log 2023-08-26T18:17:53.536 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.130569.log 2023-08-26T18:17:53.537 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.122777.log 2023-08-26T18:17:53.537 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.132834.log 2023-08-26T18:17:53.537 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.135630.log 2023-08-26T18:17:53.538 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.116881.log 2023-08-26T18:17:53.538 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.126283.log 2023-08-26T18:17:53.539 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.113045.log 2023-08-26T18:17:53.539 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.140382.log 2023-08-26T18:17:53.540 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.125443.log 2023-08-26T18:17:53.540 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.126097.log 2023-08-26T18:17:53.541 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.120756.log 2023-08-26T18:17:53.541 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.116574.log 2023-08-26T18:17:53.542 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.113330.log 2023-08-26T18:17:53.542 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.117228.log 2023-08-26T18:17:53.543 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.109752.log 2023-08-26T18:17:53.543 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.137240.log 2023-08-26T18:17:53.543 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.108666.log 2023-08-26T18:17:53.544 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.121984.log 2023-08-26T18:17:53.545 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.111108.log 2023-08-26T18:17:53.545 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.114819.log 2023-08-26T18:17:53.546 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.112150.log 2023-08-26T18:17:53.546 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.126934.log 2023-08-26T18:17:53.547 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.108616.log 2023-08-26T18:17:53.547 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.111156.log 2023-08-26T18:17:53.548 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.136767.log 2023-08-26T18:17:53.548 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.127316.log 2023-08-26T18:17:53.548 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.127933.log 2023-08-26T18:17:53.549 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.0.108093.log 2023-08-26T18:17:53.549 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.106850.log 2023-08-26T18:17:53.554 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.130282.log 2023-08-26T18:17:53.555 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.124003.log 2023-08-26T18:17:53.555 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2.audit.log 2023-08-26T18:17:53.555 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.118085.log 2023-08-26T18:17:53.556 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.123759.log 2023-08-26T18:17:53.556 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.135050.log 2023-08-26T18:17:53.557 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.138680.log 2023-08-26T18:17:53.557 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.124437.log 2023-08-26T18:17:53.558 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.136880.log 2023-08-26T18:17:53.558 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.132064.log 2023-08-26T18:17:53.559 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.134156.log 2023-08-26T18:17:53.560 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.136664.log 2023-08-26T18:17:53.561 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.112346.log 2023-08-26T18:17:53.561 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.114136.log 2023-08-26T18:17:53.561 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.131995.log 2023-08-26T18:17:53.561 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.135442.log 2023-08-26T18:17:53.562 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.126075.log 2023-08-26T18:17:53.562 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.129534.log 2023-08-26T18:17:53.563 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.119642.log 2023-08-26T18:17:53.563 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.144955.log 2023-08-26T18:17:53.563 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.129030.log 2023-08-26T18:17:53.564 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.129236.log 2023-08-26T18:17:53.564 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.123551.log 2023-08-26T18:17:53.565 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.144370.log 2023-08-26T18:17:53.565 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.123807.log 2023-08-26T18:17:53.566 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.121437.log 2023-08-26T18:17:53.566 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.3.118341.log 2023-08-26T18:17:53.567 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.120851.log 2023-08-26T18:17:53.567 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.118568.log 2023-08-26T18:17:53.568 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.114457.log 2023-08-26T18:17:53.568 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.112897.log 2023-08-26T18:17:53.568 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.132217.log 2023-08-26T18:17:53.569 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.114557.log 2023-08-26T18:17:53.570 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.126779.log 2023-08-26T18:17:53.570 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.143117.log 2023-08-26T18:17:53.574 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.129351.log 2023-08-26T18:17:53.575 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.130658.log 2023-08-26T18:17:53.576 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.119304.log 2023-08-26T18:17:53.580 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster2-client.mirror.114637.log: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.114637.log.gz 2023-08-26T18:17:53.580 INFO:teuthology.orchestra.run.smithi124.stderr: 92.0% -- replaced with /var/log/ceph/cluster1-client.mirror.0.114286.log.gz 2023-08-26T18:17:53.581 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster2-client.mirror.117740.log: 87.4% -- replaced with /var/log/ceph/cluster2-client.mirror.117740.log.gz 2023-08-26T18:17:53.581 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster2-client.mirror.136313.log: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.136313.log.gz 2023-08-26T18:17:53.581 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster2-client.mirror.136623.log: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.136623.log.gz 2023-08-26T18:17:53.582 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster1-client.mirror.141985.log: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.141985.log.gz 2023-08-26T18:17:53.582 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster2-client.mirror.115870.log: 93.7% -- replaced with /var/log/ceph/cluster2-client.mirror.118489.log.gz 2023-08-26T18:17:53.585 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster2-client.mirror.119304.log: 87.4% -- replaced with /var/log/ceph/cluster2-client.mirror.119304.log.gz 2023-08-26T18:17:53.585 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.115828.log 2023-08-26T18:17:53.586 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster2-client.mirror.115828.log: 89.7% -- replaced with /var/log/ceph/cluster2-client.mirror.115828.log.gz 2023-08-26T18:17:53.586 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.108456.log 2023-08-26T18:17:53.587 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster1-client.mirror.108456.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.108456.log.gz 2023-08-26T18:17:53.587 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.116126.log 2023-08-26T18:17:53.598 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster2-client.mirror.116126.log: /var/log/ceph/cluster2-client.mirror.114926.log: 87.4% -- replaced with /var/log/ceph/cluster2-client.mirror.114926.log.gz 2023-08-26T18:17:53.598 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.134532.log 2023-08-26T18:17:53.611 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster1-client.mirror.126488.log: 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.126488.log.gz 2023-08-26T18:17:53.612 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster2-client.mirror.106082.log: /var/log/ceph/cluster2-client.mirror.136540.log: 88.0% -- replaced with /var/log/ceph/cluster2-client.mirror.136540.log.gz 2023-08-26T18:17:53.612 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster1-client.mirror.131787.log: 87.2% -- replaced with /var/log/ceph/cluster1-client.mirror.131787.log.gz 2023-08-26T18:17:53.612 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster2-client.mirror.134678.log: 88.0% -- replaced with /var/log/ceph/cluster2-client.mirror.134678.log.gz 2023-08-26T18:17:53.613 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster2-client.mirror.114717.log: 88.1% -- replaced with /var/log/ceph/cluster2-client.mirror.114717.log.gz 2023-08-26T18:17:53.614 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster2-client.mirror.134532.log: /var/log/ceph/cluster2-client.mirror.116183.log: 87.5% -- replaced with /var/log/ceph/cluster2-client.mirror.116183.log.gz 2023-08-26T18:17:53.614 INFO:teuthology.orchestra.run.smithi124.stderr: 88.0% -- replaced with /var/log/ceph/cluster2-client.mirror.134532.log.gz 2023-08-26T18:17:53.615 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster1-client.mirror.130569.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.134218.log 2023-08-26T18:17:53.615 INFO:teuthology.orchestra.run.smithi124.stderr: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.130569.log.gz 2023-08-26T18:17:53.615 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster2-client.mirror.122777.log: 87.4% -- replaced with /var/log/ceph/cluster2-client.mirror.122777.log.gz 2023-08-26T18:17:53.623 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster2-client.mirror.132834.log: 89.7% -- replaced with /var/log/ceph/cluster2-client.mirror.132834.log.gz 2023-08-26T18:17:53.624 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster2-client.mirror.135630.log: 90.5% -- replaced with /var/log/ceph/cluster2-client.mirror.135630.log.gz 2023-08-26T18:17:53.624 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster2-client.mirror.116881.log: 89.6% -- replaced with /var/log/ceph/cluster2-client.mirror.116881.log.gz 2023-08-26T18:17:53.624 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster2-client.mirror.126283.log: 92.6% -- replaced with /var/log/ceph/cluster2-client.mirror.126283.log.gz 2023-08-26T18:17:53.625 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster2-client.mirror.113045.log: 89.2% -- replaced with /var/log/ceph/cluster2-client.mirror.113045.log.gz 2023-08-26T18:17:53.625 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster1-client.mirror.140382.log: 86.8% -- replaced with /var/log/ceph/cluster1-client.mirror.140382.log.gz 2023-08-26T18:17:53.626 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster1-client.mirror.125443.log: 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.125443.log.gz 2023-08-26T18:17:53.626 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster2-client.mirror.126097.log: 89.7% -- replaced with /var/log/ceph/cluster2-client.mirror.126097.log.gz 2023-08-26T18:17:53.633 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster2-client.mirror.120756.log: /var/log/ceph/cluster2-client.mirror.134218.log: 92.4% -- replaced with /var/log/ceph/cluster2-client.mirror.120756.log.gz 2023-08-26T18:17:53.633 INFO:teuthology.orchestra.run.smithi124.stderr: 88.0% -- replaced with /var/log/ceph/cluster2-client.mirror.134218.log.gz 2023-08-26T18:17:53.633 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.127477.log 2023-08-26T18:17:53.644 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster2-client.mirror.116574.log: /var/log/ceph/cluster1-client.mirror.127477.log: 93.7% -- replaced with /var/log/ceph/cluster2-client.mirror.118301.log.gz 2023-08-26T18:17:53.644 INFO:teuthology.orchestra.run.smithi124.stderr: 92.3% -- replaced with /var/log/ceph/cluster1-client.mirror.127477.log.gz 2023-08-26T18:17:53.644 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.110590.log 2023-08-26T18:17:53.645 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster2-client.mirror.113330.log: 90.1% -- replaced with /var/log/ceph/cluster2-client.mirror.113330.log.gz 2023-08-26T18:17:53.645 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster2-client.mirror.117228.log: 87.4% -- replaced with /var/log/ceph/cluster2-client.mirror.117228.log.gz 2023-08-26T18:17:53.645 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster1-client.mirror.109752.log: 87.9% -- replaced with /var/log/ceph/cluster1-client.mirror.109752.log.gz 2023-08-26T18:17:53.646 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster1-client.mirror.137240.log: 86.8% -- replaced with /var/log/ceph/cluster1-client.mirror.137240.log.gz 2023-08-26T18:17:53.646 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster2-client.mirror.108666.log: 89.7% -- replaced with /var/log/ceph/cluster2-client.mirror.108666.log.gz 2023-08-26T18:17:53.647 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster1-client.mirror.121984.log: 92.2% -- replaced with /var/log/ceph/cluster1-client.mirror.121984.log.gz 2023-08-26T18:17:53.648 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster2-client.mirror.111108.log: 91.7% -- replaced with /var/log/ceph/cluster2-client.mirror.111108.log.gz 2023-08-26T18:17:53.658 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster2-client.mirror.114819.log: 90.9% -- replaced with /var/log/ceph/cluster2-client.mirror.114819.log.gz 2023-08-26T18:17:53.658 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster1-client.mirror.112150.log: 87.8% -- replaced with /var/log/ceph/cluster1-client.mirror.112150.log.gz 2023-08-26T18:17:53.658 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster1-client.mirror.126934.log: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.126934.log.gz 2023-08-26T18:17:53.659 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster2-client.mirror.108616.log: /var/log/ceph/cluster2-client.mirror.110590.log: 91.6% -- replaced with /var/log/ceph/cluster2-client.mirror.110590.log.gz 2023-08-26T18:17:53.659 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.110921.log 2023-08-26T18:17:53.659 INFO:teuthology.orchestra.run.smithi124.stderr: 93.4% -- replaced with /var/log/ceph/cluster2-client.mirror.108616.log.gz 2023-08-26T18:17:53.670 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster2-client.mirror.111156.log: /var/log/ceph/cluster2-client.mirror.110921.log: 93.9% -- replaced with /var/log/ceph/cluster2-client.mirror.111156.log.gz 2023-08-26T18:17:53.672 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster2-client.mirror.136767.log: 90.4% -- replaced with /var/log/ceph/cluster2-client.mirror.136767.log.gz 2023-08-26T18:17:53.672 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster2-client.mirror.127316.log: 87.3% -- replaced with /var/log/ceph/cluster2-client.mirror.127316.log.gz 2023-08-26T18:17:53.672 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster2-client.mirror.127933.log: 92.6% -- replaced with /var/log/ceph/cluster2-client.mirror.127933.log.gz 2023-08-26T18:17:53.673 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster2-client.mirror.0.108093.log: 93.4% -- replaced with /var/log/ceph/cluster2-client.mirror.110921.log.gz 2023-08-26T18:17:53.673 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.142294.log 2023-08-26T18:17:53.680 INFO:teuthology.orchestra.run.smithi124.stderr: 93.8% -- replaced with /var/log/ceph/cluster2-client.mirror.0.108093.log.gz 2023-08-26T18:17:53.680 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster2-client.mirror.106850.log: 55.8% -- replaced with /var/log/ceph/cluster2-client.mirror.106850.log.gz 2023-08-26T18:17:53.680 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster1-client.mirror.130282.log: 87.6% -- replaced with /var/log/ceph/cluster1-client.mirror.130282.log.gz 2023-08-26T18:17:53.681 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster1-client.mirror.124003.log: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.124003.log.gz 2023-08-26T18:17:53.681 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster2.audit.log: 92.7% -- replaced with /var/log/ceph/cluster2.audit.log.gz 2023-08-26T18:17:53.681 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster2-client.mirror.118085.log: 89.6% -- replaced with /var/log/ceph/cluster2-client.mirror.118085.log.gz 2023-08-26T18:17:53.682 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster1-client.mirror.123759.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.123759.log.gz 2023-08-26T18:17:53.682 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster2-client.mirror.135050.log: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.135050.log.gz 2023-08-26T18:17:53.682 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster1-client.mirror.138680.log: 86.8% -- replaced with /var/log/ceph/cluster1-client.mirror.138680.log.gz 2023-08-26T18:17:53.691 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster2-client.mirror.124437.log: 89.8% -- replaced with /var/log/ceph/cluster2-client.mirror.124437.log.gz 2023-08-26T18:17:53.691 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster1-client.mirror.136880.log: /var/log/ceph/cluster1-client.mirror.142294.log: 86.8% -- replaced with /var/log/ceph/cluster1-client.mirror.136880.log.gz 2023-08-26T18:17:53.692 INFO:teuthology.orchestra.run.smithi124.stderr: 86.8% -- replaced with /var/log/ceph/cluster1-client.mirror.142294.log.gz 2023-08-26T18:17:53.692 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster2-client.mirror.132064.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.142968.log 2023-08-26T18:17:53.692 INFO:teuthology.orchestra.run.smithi124.stderr: 92.6% -- replaced with /var/log/ceph/cluster2-client.mirror.132064.log.gz 2023-08-26T18:17:53.692 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster2-client.mirror.134156.log: 87.9% -- replaced with /var/log/ceph/cluster2-client.mirror.134156.log.gz 2023-08-26T18:17:53.693 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster2-client.mirror.136664.log: 88.0% -- replaced with /var/log/ceph/cluster2-client.mirror.136664.log.gz 2023-08-26T18:17:53.693 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster1-client.mirror.112346.log: 89.1% -- replaced with /var/log/ceph/cluster1-client.mirror.112346.log.gz 2023-08-26T18:17:53.693 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster1-client.mirror.114136.log: 93.4% -- replaced with /var/log/ceph/cluster1-client.mirror.114136.log.gz 2023-08-26T18:17:53.693 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster1-client.mirror.131995.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.131995.log.gz 2023-08-26T18:17:53.698 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster2-client.mirror.135442.log: 90.5% -- replaced with /var/log/ceph/cluster2-client.mirror.135442.log.gz 2023-08-26T18:17:53.698 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster1-client.mirror.126075.log: 92.2% -- replaced with /var/log/ceph/cluster1-client.mirror.126075.log.gz 2023-08-26T18:17:53.698 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster2-client.mirror.129534.log: 89.8% -- replaced with /var/log/ceph/cluster2-client.mirror.129534.log.gz 2023-08-26T18:17:53.699 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster2-client.mirror.119642.log: 89.6% -- replaced with /var/log/ceph/cluster2-client.mirror.119642.log.gz 2023-08-26T18:17:53.699 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster2-client.mirror.144955.log: 7.1% -- replaced with /var/log/ceph/cluster2-client.mirror.144955.log.gz 2023-08-26T18:17:53.699 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster1-client.mirror.129030.log: 92.3% -- replaced with /var/log/ceph/cluster1-client.mirror.129030.log.gz 2023-08-26T18:17:53.710 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster2-client.mirror.129236.log: 92.3% -- replaced with /var/log/ceph/cluster2-client.mirror.129236.log.gz 2023-08-26T18:17:53.710 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster1-client.mirror.123551.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.123551.log.gz 2023-08-26T18:17:53.710 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster2-client.mirror.144370.log: 7.1% -- replaced with /var/log/ceph/cluster2-client.mirror.144370.log.gz 2023-08-26T18:17:53.710 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster1-client.mirror.123807.log: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.123807.log.gz 2023-08-26T18:17:53.711 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster1-client.mirror.121437.log: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.121437.log.gz 2023-08-26T18:17:53.711 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster2-client.mirror.3.118341.log: /var/log/ceph/cluster1-client.mirror.142968.log: 68.1% -- replaced with /var/log/ceph/cluster1-client.mirror.142968.log.gz 2023-08-26T18:17:53.711 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.142784.log 2023-08-26T18:17:53.721 INFO:teuthology.orchestra.run.smithi124.stderr: 94.1% -- replaced with /var/log/ceph/cluster2-client.mirror.106082.log.gz 2023-08-26T18:17:53.721 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster2-client.mirror.120851.log: 89.8% -- replaced with /var/log/ceph/cluster2-client.mirror.120851.log.gz 2023-08-26T18:17:53.721 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster2-client.mirror.118568.log: /var/log/ceph/cluster1-client.mirror.142784.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.142784.log.gz 2023-08-26T18:17:53.722 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.123253.log 2023-08-26T18:17:53.722 INFO:teuthology.orchestra.run.smithi124.stderr: 93.7% -- replaced with /var/log/ceph/cluster2-client.mirror.140532.log.gz 2023-08-26T18:17:53.723 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster2-client.mirror.114457.log: 88.5% -- replaced with /var/log/ceph/cluster2-client.mirror.114457.log.gz 2023-08-26T18:17:53.723 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster1-client.mirror.112897.log: 74.7% -- replaced with /var/log/ceph/cluster1-client.mirror.112897.log.gz 2023-08-26T18:17:53.727 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster1-client.mirror.132217.log: 87.2% -- replaced with /var/log/ceph/cluster1-client.mirror.132217.log.gz 2023-08-26T18:17:53.727 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster2-client.mirror.114557.log: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.114557.log.gz 2023-08-26T18:17:53.728 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster1-client.mirror.126779.log: /var/log/ceph/cluster2-client.mirror.123253.log: 87.4% -- replaced with /var/log/ceph/cluster2-client.mirror.123253.log.gz 2023-08-26T18:17:53.729 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.109632.log 2023-08-26T18:17:53.729 INFO:teuthology.orchestra.run.smithi124.stderr: 92.3% -- replaced with /var/log/ceph/cluster1-client.mirror.126779.log.gz 2023-08-26T18:17:53.729 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster1-client.mirror.143117.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.143117.log.gz 2023-08-26T18:17:53.730 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster2-client.mirror.129351.log: 87.3% -- replaced with /var/log/ceph/cluster2-client.mirror.129351.log.gz 2023-08-26T18:17:53.730 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster1-client.mirror.130658.log: 92.3% -- replaced with /var/log/ceph/cluster1-client.mirror.130658.log.gz 2023-08-26T18:17:53.739 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster2-client.mirror.109632.log: 88.1% -- replaced with /var/log/ceph/cluster2-client.mirror.109632.log.gz 2023-08-26T18:17:53.740 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.119890.log 2023-08-26T18:17:53.761 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster1-client.mirror.119890.log: 95.4% -- replaced with /var/log/ceph/cluster1-client.mirror.119890.log.gz 2023-08-26T18:17:53.761 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.143485.log 2023-08-26T18:17:53.772 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.120919.log 2023-08-26T18:17:53.783 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster1-client.mirror.120919.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.120919.log.gz 2023-08-26T18:17:53.783 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.140750.log 2023-08-26T18:17:53.783 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster2-client.admin.143485.log: 9.2% -- replaced with /var/log/ceph/cluster2-client.admin.143485.log.gz 2023-08-26T18:17:53.794 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster2-client.mirror.140750.log: 89.7% -- replaced with /var/log/ceph/cluster2-client.mirror.140750.log.gz 2023-08-26T18:17:53.794 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.141964.log 2023-08-26T18:17:53.805 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster2-client.mirror.141964.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.122184.log 2023-08-26T18:17:53.805 INFO:teuthology.orchestra.run.smithi124.stderr: 88.6% -- replaced with /var/log/ceph/cluster2-client.mirror.141964.log.gz 2023-08-26T18:17:53.811 INFO:teuthology.orchestra.run.smithi124.stderr: 93.7% -- replaced with /var/log/ceph/cluster2-client.mirror.115870.log.gz 2023-08-26T18:17:53.811 INFO:teuthology.orchestra.run.smithi124.stderr: 93.8% -- replaced with /var/log/ceph/cluster2-client.mirror.109370.log.gz 2023-08-26T18:17:53.813 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster2-client.mirror.122184.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.139063.log 2023-08-26T18:17:53.813 INFO:teuthology.orchestra.run.smithi124.stderr: 92.4% -- replaced with /var/log/ceph/cluster2-client.mirror.122184.log.gz 2023-08-26T18:17:53.814 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster1-client.mirror.139063.log: 86.8% -- replaced with /var/log/ceph/cluster1-client.mirror.139063.log.gz 2023-08-26T18:17:53.815 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.134760.log 2023-08-26T18:17:53.818 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster2-client.mirror.134760.log: 90.5% -- replaced with /var/log/ceph/cluster2-client.mirror.134760.log.gz 2023-08-26T18:17:53.819 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.119030.log 2023-08-26T18:17:53.820 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster2-client.mirror.119030.log: 79.5% -- replaced with /var/log/ceph/cluster2-client.mirror.119030.log.gz 2023-08-26T18:17:53.820 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.114116.log 2023-08-26T18:17:53.825 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster2-client.mirror.114116.log: 94.6% -- replaced with /var/log/ceph/cluster2-client.mirror.3.118341.log.gz 2023-08-26T18:17:53.826 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.111890.log 2023-08-26T18:17:53.826 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster1-client.mirror.111890.log: 88.8% -- replaced with /var/log/ceph/cluster1-client.mirror.111890.log.gz 2023-08-26T18:17:53.827 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.125877.log 2023-08-26T18:17:53.827 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster1-client.mirror.125877.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.125877.log.gz 2023-08-26T18:17:53.827 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.143249.log 2023-08-26T18:17:53.828 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster1-client.mirror.143249.log: 2.3% -- replaced with /var/log/ceph/cluster1-client.mirror.143249.log.gz 2023-08-26T18:17:53.828 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.133032.log 2023-08-26T18:17:53.828 INFO:teuthology.orchestra.run.smithi124.stderr: 93.4% -- replaced with /var/log/ceph/cluster2-client.mirror.114116.log.gz 2023-08-26T18:17:53.829 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster1-client.mirror.133032.log: 92.2% -- replaced with /var/log/ceph/cluster1-client.mirror.133032.log.gz 2023-08-26T18:17:53.830 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.131874.log 2023-08-26T18:17:53.830 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster2-client.mirror.131874.log: 89.7% -- replaced with /var/log/ceph/cluster2-client.mirror.131874.log.gz 2023-08-26T18:17:53.830 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.120735.log 2023-08-26T18:17:53.831 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.132265.log 2023-08-26T18:17:53.832 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster1-client.mirror.120735.log: 87.6% -- replaced with /var/log/ceph/cluster1-client.mirror.120735.log.gz 2023-08-26T18:17:53.833 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.142029.log 2023-08-26T18:17:53.833 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster1-client.mirror.132265.log: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.132265.log.gz 2023-08-26T18:17:53.834 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.123301.log 2023-08-26T18:17:53.835 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster2-client.mirror.142029.log: 88.1% -- replaced with /var/log/ceph/cluster2-client.mirror.142029.log.gz 2023-08-26T18:17:53.835 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster1-client.mirror.123301.log: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.123301.log.gz 2023-08-26T18:17:53.835 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.128271.log 2023-08-26T18:17:53.836 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.117119.log 2023-08-26T18:17:53.836 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster1-client.mirror.128271.log: 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.128271.log.gz 2023-08-26T18:17:53.846 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster2-client.mirror.117119.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.133241.log 2023-08-26T18:17:53.847 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster1-client.mirror.133241.log: 87.6% -- replaced with /var/log/ceph/cluster1-client.mirror.133241.log.gz 2023-08-26T18:17:53.858 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.121847.log 2023-08-26T18:17:53.869 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster1-client.mirror.121847.log: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.121847.log.gz 2023-08-26T18:17:53.869 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.114738.log 2023-08-26T18:17:53.882 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster2-client.mirror.114738.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.142933.log 2023-08-26T18:17:53.883 INFO:teuthology.orchestra.run.smithi124.stderr: 87.9% -- replaced with /var/log/ceph/cluster2-client.mirror.114738.log.gz 2023-08-26T18:17:53.894 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster1-client.mirror.142933.log: 48.2% -- replaced with /var/log/ceph/cluster1-client.mirror.142933.log.gz 2023-08-26T18:17:53.894 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.142711.log 2023-08-26T18:17:53.894 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster1-client.mirror.142711.log: 55.8% -- replaced with /var/log/ceph/cluster1-client.mirror.142711.log.gz 2023-08-26T18:17:53.895 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.138005.log 2023-08-26T18:17:53.895 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster1-client.mirror.138005.log: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.138005.log.gz 2023-08-26T18:17:53.895 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.117841.log 2023-08-26T18:17:53.896 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster2-client.mirror.117841.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.0.106173.log 2023-08-26T18:17:53.906 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster2-client.mirror.0.106173.log: 91.6% -- replaced with /var/log/ceph/cluster2-client.mirror.0.106173.log.gz 2023-08-26T18:17:53.907 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.123979.log 2023-08-26T18:17:53.919 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster2-client.mirror.123979.log: 87.3% -- replaced with /var/log/ceph/cluster2-client.mirror.123979.log.gz 2023-08-26T18:17:53.923 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.112120.log 2023-08-26T18:17:53.923 INFO:teuthology.orchestra.run.smithi124.stderr: 93.7% -- replaced with /var/log/ceph/cluster2-client.mirror.116126.log.gz 2023-08-26T18:17:53.924 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster1-client.admin.112120.log: 88.0% -- replaced with /var/log/ceph/cluster1-client.admin.112120.log.gz 2023-08-26T18:17:53.924 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.135029.log 2023-08-26T18:17:53.924 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster2-client.mirror.135029.log: 88.1% -- replaced with /var/log/ceph/cluster2-client.mirror.135029.log.gz 2023-08-26T18:17:53.925 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.122403.log 2023-08-26T18:17:53.925 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster1-client.mirror.122403.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.122403.log.gz 2023-08-26T18:17:53.925 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.1.106205.log 2023-08-26T18:17:53.931 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster2-client.mirror.1.106205.log: 92.3% -- replaced with /var/log/ceph/cluster2-client.mirror.1.106205.log.gz 2023-08-26T18:17:53.931 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.144474.log 2023-08-26T18:17:53.931 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster2-client.mirror.144474.log: 8.2% -- replaced with /var/log/ceph/cluster2-client.mirror.144474.log.gz 2023-08-26T18:17:53.932 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.135692.log 2023-08-26T18:17:53.932 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster2-client.mirror.135692.log: 90.4% -- replaced with /var/log/ceph/cluster2-client.mirror.135692.log.gz 2023-08-26T18:17:53.932 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.139159.log 2023-08-26T18:17:53.933 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster2-client.mirror.139159.log: 87.5% -- replaced with /var/log/ceph/cluster2-client.mirror.139159.log.gz 2023-08-26T18:17:53.933 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.138545.log 2023-08-26T18:17:53.934 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster1-client.mirror.138545.log: 86.7% -- replaced with /var/log/ceph/cluster1-client.mirror.138545.log.gz 2023-08-26T18:17:53.934 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.142694.log 2023-08-26T18:17:53.935 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster1-client.mirror.142694.log: 66.3% -- replaced with /var/log/ceph/cluster1-client.mirror.142694.log.gz 2023-08-26T18:17:53.935 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.109138.log 2023-08-26T18:17:53.935 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster1-client.mirror.109138.log: 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.109138.log.gz 2023-08-26T18:17:53.936 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.105008.log 2023-08-26T18:17:53.936 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster2-client.admin.105008.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.105008.log.gz 2023-08-26T18:17:53.937 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.125226.log 2023-08-26T18:17:53.938 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster2-client.mirror.125226.log: 92.6% -- replaced with /var/log/ceph/cluster2-client.mirror.125226.log.gz 2023-08-26T18:17:53.939 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.137600.log 2023-08-26T18:17:53.939 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster1-client.mirror.137600.log: 86.6% -- replaced with /var/log/ceph/cluster1-client.mirror.137600.log.gz 2023-08-26T18:17:53.940 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.112506.log 2023-08-26T18:17:53.945 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster2-client.mirror.112506.log: 93.7% -- replaced with /var/log/ceph/cluster2-client.mirror.118568.log.gz 2023-08-26T18:17:53.945 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.119741.log 2023-08-26T18:17:53.960 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster2-client.mirror.119741.log: 87.4% -- replaced with /var/log/ceph/cluster2-client.mirror.119741.log.gz 2023-08-26T18:17:53.964 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.109680.log 2023-08-26T18:17:53.964 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster1-client.mirror.109680.log: 88.1% -- replaced with /var/log/ceph/cluster1-client.mirror.109680.log.gz 2023-08-26T18:17:53.964 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.122688.log 2023-08-26T18:17:53.965 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster1-client.mirror.122688.log: 92.3% -- replaced with /var/log/ceph/cluster1-client.mirror.122688.log.gz 2023-08-26T18:17:53.965 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.143155.log 2023-08-26T18:17:53.965 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster1-client.mirror.143155.log: 2.3% -- replaced with /var/log/ceph/cluster1-client.mirror.143155.log.gz 2023-08-26T18:17:53.966 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.120776.log 2023-08-26T18:17:53.967 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster1-client.mirror.120776.log: 92.1% -- replaced with /var/log/ceph/cluster1-client.mirror.120776.log.gz 2023-08-26T18:17:53.967 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.2.112546.log 2023-08-26T18:17:53.967 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster1-client.mirror.2.112546.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.121943.log 2023-08-26T18:17:53.968 INFO:teuthology.orchestra.run.smithi124.stderr: 93.7% -- replaced with /var/log/ceph/cluster2-client.mirror.116574.log.gz 2023-08-26T18:17:53.969 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster1-client.mirror.121943.log: 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.121943.log.gz 2023-08-26T18:17:53.977 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.134573.log 2023-08-26T18:17:53.980 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster2-client.mirror.134573.log: 90.4% -- replaced with /var/log/ceph/cluster2-client.mirror.134573.log.gz 2023-08-26T18:17:53.980 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.131354.log 2023-08-26T18:17:53.993 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster2-client.mirror.131354.log: 89.7% -- replaced with /var/log/ceph/cluster2-client.mirror.131354.log.gz 2023-08-26T18:17:53.993 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.123783.log 2023-08-26T18:17:53.994 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster1-client.mirror.123783.log: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.123783.log.gz 2023-08-26T18:17:53.994 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.124710.log 2023-08-26T18:17:53.995 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster1-client.mirror.124710.log: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.124710.log.gz 2023-08-26T18:17:53.995 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.134003.log 2023-08-26T18:17:53.998 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster2-client.mirror.134003.log: 90.4% -- replaced with /var/log/ceph/cluster2-client.mirror.134003.log.gz 2023-08-26T18:17:53.998 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.0.115722.log 2023-08-26T18:17:54.010 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster1-client.mirror.0.115722.log: 93.2% -- replaced with /var/log/ceph/cluster1-client.mirror.2.112546.log.gz 2023-08-26T18:17:54.010 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.120895.log 2023-08-26T18:17:54.011 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster1-client.mirror.120895.log: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.120895.log.gz 2023-08-26T18:17:54.011 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.111938.log 2023-08-26T18:17:54.012 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster1-client.mirror.111938.log: 88.2% -- replaced with /var/log/ceph/cluster1-client.mirror.111938.log.gz 2023-08-26T18:17:54.013 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.132435.log 2023-08-26T18:17:54.016 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster2-client.mirror.132435.log: 89.8% -- replaced with /var/log/ceph/cluster2-client.mirror.132435.log.gz 2023-08-26T18:17:54.017 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.113858.log 2023-08-26T18:17:54.017 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster2-client.mirror.113858.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.142644.log 2023-08-26T18:17:54.017 INFO:teuthology.orchestra.run.smithi124.stderr: 93.4% -- replaced with /var/log/ceph/cluster2-client.mirror.113858.log.gz 2023-08-26T18:17:54.018 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster1-client.admin.142644.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.142644.log.gz 2023-08-26T18:17:54.018 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.142108.log 2023-08-26T18:17:54.018 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster2-client.mirror.142108.log: 89.6% -- replaced with /var/log/ceph/cluster2-client.mirror.142108.log.gz 2023-08-26T18:17:54.019 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.135990.log 2023-08-26T18:17:54.019 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster2-client.mirror.135990.log: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.135990.log.gz 2023-08-26T18:17:54.020 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.116594.log 2023-08-26T18:17:54.020 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster2-client.mirror.116594.log: 79.4% -- replaced with /var/log/ceph/cluster2-client.mirror.116594.log.gz 2023-08-26T18:17:54.021 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.114577.log 2023-08-26T18:17:54.021 INFO:teuthology.orchestra.run.smithi124.stderr: 92.8% -- replaced with /var/log/ceph/cluster1-client.mirror.0.115722.log.gz 2023-08-26T18:17:54.021 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.131947.log 2023-08-26T18:17:54.022 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster2-client.mirror.114577.log: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.114577.log.gz 2023-08-26T18:17:54.022 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster2-client.mirror.131947.log: 87.4% -- replaced with /var/log/ceph/cluster2-client.mirror.131947.log.gz 2023-08-26T18:17:54.022 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.2.106082.log 2023-08-26T18:17:54.033 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster1-client.mirror.2.106082.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.122531.log 2023-08-26T18:17:54.034 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster2-client.mirror.122531.log: 89.8% -- replaced with /var/log/ceph/cluster2-client.mirror.122531.log.gz 2023-08-26T18:17:54.044 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.112002.log 2023-08-26T18:17:54.056 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster2-client.mirror.112002.log: 93.4% -- replaced with /var/log/ceph/cluster2-client.mirror.112002.log.gz 2023-08-26T18:17:54.057 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.114758.log 2023-08-26T18:17:54.057 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster2-client.mirror.114758.log: 88.0% -- replaced with /var/log/ceph/cluster2-client.mirror.114758.log.gz 2023-08-26T18:17:54.058 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.111370.log 2023-08-26T18:17:54.059 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster1-client.mirror.111370.log: 88.6% -- replaced with /var/log/ceph/cluster1-client.mirror.111370.log.gz 2023-08-26T18:17:54.059 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.134239.log 2023-08-26T18:17:54.060 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster2-client.mirror.134239.log: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.134239.log.gz 2023-08-26T18:17:54.060 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.137488.log 2023-08-26T18:17:54.060 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster1-client.mirror.137488.log: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.137488.log.gz 2023-08-26T18:17:54.061 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.134946.log 2023-08-26T18:17:54.062 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster2-client.mirror.134946.log: 90.4% -- replaced with /var/log/ceph/cluster2-client.mirror.134946.log.gz 2023-08-26T18:17:54.062 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.128175.log 2023-08-26T18:17:54.063 INFO:teuthology.orchestra.run.smithi124.stderr: 93.7% -- replaced with /var/log/ceph/cluster2-client.mirror.117119.log.gz 2023-08-26T18:17:54.063 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster1-client.mirror.128175.log: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.128175.log.gz 2023-08-26T18:17:54.064 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.135940.log 2023-08-26T18:17:54.065 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster2-client.mirror.135940.log: 90.5% -- replaced with /var/log/ceph/cluster2-client.mirror.135940.log.gz 2023-08-26T18:17:54.065 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.121663.log 2023-08-26T18:17:54.067 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster1-client.mirror.121663.log: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.121663.log.gz 2023-08-26T18:17:54.067 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.131811.log 2023-08-26T18:17:54.067 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.131446.log 2023-08-26T18:17:54.068 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster1-client.mirror.131811.log: 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.131811.log.gz 2023-08-26T18:17:54.068 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster1-client.mirror.131446.log: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.131446.log.gz 2023-08-26T18:17:54.069 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.125181.log 2023-08-26T18:17:54.069 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster1-client.mirror.125181.log: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.125181.log.gz 2023-08-26T18:17:54.070 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.112462.log 2023-08-26T18:17:54.073 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster1-client.mirror.112462.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.128655.log 2023-08-26T18:17:54.074 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster2-client.mirror.128655.log: 87.4% -- replaced with /var/log/ceph/cluster2-client.mirror.128655.log.gz 2023-08-26T18:17:54.074 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.141573.log 2023-08-26T18:17:54.076 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster2-client.mirror.141573.log: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.141573.log.gz 2023-08-26T18:17:54.076 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.124096.log 2023-08-26T18:17:54.076 INFO:teuthology.orchestra.run.smithi124.stderr: 93.4% -- replaced with /var/log/ceph/cluster1-client.mirror.112462.log.gz 2023-08-26T18:17:54.077 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster2-client.mirror.124096.log: 92.5% -- replaced with /var/log/ceph/cluster2-client.mirror.124096.log.gz 2023-08-26T18:17:54.078 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.118733.log 2023-08-26T18:17:54.078 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster2-client.mirror.118733.log: 87.5% -- replaced with /var/log/ceph/cluster2-client.mirror.118733.log.gz 2023-08-26T18:17:54.079 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.137578.log 2023-08-26T18:17:54.079 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster1-client.mirror.137578.log: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.137578.log.gz 2023-08-26T18:17:54.080 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.119435.log 2023-08-26T18:17:54.081 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster2-client.mirror.119435.log: 79.5% -- replaced with /var/log/ceph/cluster2-client.mirror.119435.log.gz 2023-08-26T18:17:54.081 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.115013.log 2023-08-26T18:17:54.082 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster2-client.mirror.115013.log: 79.4% -- replaced with /var/log/ceph/cluster2-client.mirror.115013.log.gz 2023-08-26T18:17:54.082 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.105169.log 2023-08-26T18:17:54.084 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster2-client.admin.105169.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.105169.log.gz 2023-08-26T18:17:54.085 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.138478.log 2023-08-26T18:17:54.085 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster1-client.mirror.138478.log: 86.8% -- replaced with /var/log/ceph/cluster1-client.mirror.138478.log.gz 2023-08-26T18:17:54.085 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.127138.log 2023-08-26T18:17:54.086 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster1-client.mirror.127138.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.127138.log.gz 2023-08-26T18:17:54.086 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.138185.log 2023-08-26T18:17:54.086 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster1-client.mirror.138185.log: 86.8% -- replaced with /var/log/ceph/cluster1-client.mirror.138185.log.gz 2023-08-26T18:17:54.086 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.133344.log 2023-08-26T18:17:54.087 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster2-client.mirror.133344.log: 94.5% -- replaced with /var/log/ceph/cluster1-client.mirror.2.106082.log.gz 2023-08-26T18:17:54.088 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.113428.log 2023-08-26T18:17:54.090 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster2-client.mirror.113428.log: 92.0% -- replaced with /var/log/ceph/cluster2-client.mirror.133344.log.gz 2023-08-26T18:17:54.092 INFO:teuthology.orchestra.run.smithi124.stderr: 93.4% -- replaced with /var/log/ceph/cluster2-client.mirror.113428.log.gz 2023-08-26T18:17:54.092 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.113117.log 2023-08-26T18:17:54.093 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.107908.log 2023-08-26T18:17:54.094 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster1-client.mirror.113117.log: /var/log/ceph/cluster2-client.mirror.107908.log: 89.1% -- replaced with /var/log/ceph/cluster1-client.mirror.113117.log.gz 2023-08-26T18:17:54.095 INFO:teuthology.orchestra.run.smithi124.stderr: 90.8% -- replaced with /var/log/ceph/cluster2-client.mirror.107908.log.gz 2023-08-26T18:17:54.095 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.111546.log 2023-08-26T18:17:54.096 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.113249.log 2023-08-26T18:17:54.100 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster2-client.mirror.111546.log: /var/log/ceph/cluster1-client.mirror.113249.log: 93.4% -- replaced with /var/log/ceph/cluster1-client.mirror.113249.log.gz 2023-08-26T18:17:54.100 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.142575.log 2023-08-26T18:17:54.101 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster1-client.mirror.142575.log: 86.8% -- replaced with /var/log/ceph/cluster1-client.mirror.142575.log.gz 2023-08-26T18:17:54.102 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.113944.log 2023-08-26T18:17:54.107 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster2-client.mirror.113944.log: 93.3% -- replaced with /var/log/ceph/cluster2-client.mirror.113944.log.gz 2023-08-26T18:17:54.108 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.136127.log 2023-08-26T18:17:54.109 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster2-client.mirror.136127.log: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.136127.log.gz 2023-08-26T18:17:54.109 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.127695.log 2023-08-26T18:17:54.109 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.121413.log 2023-08-26T18:17:54.110 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.116278.log 2023-08-26T18:17:54.110 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster1-client.mirror.121413.log: /var/log/ceph/cluster2-client.mirror.127695.log: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.121413.log.gz 2023-08-26T18:17:54.111 INFO:teuthology.orchestra.run.smithi124.stderr: 92.6% -- replaced with /var/log/ceph/cluster2-client.mirror.127695.log.gz 2023-08-26T18:17:54.112 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster2-client.mirror.116278.log: 87.5% -- replaced with /var/log/ceph/cluster2-client.mirror.116278.log.gz 2023-08-26T18:17:54.112 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.137645.log 2023-08-26T18:17:54.113 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster1-client.mirror.137645.log: 86.8% -- replaced with /var/log/ceph/cluster1-client.mirror.137645.log.gz 2023-08-26T18:17:54.113 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.119010.log 2023-08-26T18:17:54.119 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster2-client.mirror.119010.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.128989.log 2023-08-26T18:17:54.120 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster1-client.mirror.128989.log: 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.128989.log.gz 2023-08-26T18:17:54.120 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.134284.log 2023-08-26T18:17:54.121 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster2-client.mirror.134284.log: 87.9% -- replaced with /var/log/ceph/cluster2-client.mirror.134284.log.gz 2023-08-26T18:17:54.121 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.127388.log 2023-08-26T18:17:54.122 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster1-client.mirror.127388.log: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.127388.log.gz 2023-08-26T18:17:54.122 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.0.128059.log 2023-08-26T18:17:54.132 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster2-client.mirror.0.128059.log: 93.9% -- replaced with /var/log/ceph/cluster2-client.mirror.0.128059.log.gz 2023-08-26T18:17:54.132 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.0.108781.log 2023-08-26T18:17:54.135 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster1-client.mirror.0.108781.log: 92.0% -- replaced with /var/log/ceph/cluster1-client.mirror.0.108781.log.gz 2023-08-26T18:17:54.135 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.126372.log 2023-08-26T18:17:54.136 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster2-client.mirror.126372.log: 89.8% -- replaced with /var/log/ceph/cluster2-client.mirror.126372.log.gz 2023-08-26T18:17:54.137 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.120105.log 2023-08-26T18:17:54.138 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster2-client.mirror.120105.log: 89.7% -- replaced with /var/log/ceph/cluster2-client.mirror.120105.log.gz 2023-08-26T18:17:54.138 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.138410.log 2023-08-26T18:17:54.139 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/cluster1-client.mirror.138410.log: 86.8% -- replaced with /var/log/ceph/cluster1-client.mirror.138410.log.gz 2023-08-26T18:17:54.261 INFO:teuthology.orchestra.run.smithi124.stderr: 93.8% -- replaced with /var/log/ceph/cluster2-client.mirror.112506.log.gz 2023-08-26T18:17:54.332 INFO:teuthology.orchestra.run.smithi124.stderr: 93.7% -- replaced with /var/log/ceph/cluster2-client.mirror.117841.log.gz 2023-08-26T18:17:54.465 INFO:teuthology.orchestra.run.smithi124.stderr: 93.8% -- replaced with /var/log/ceph/cluster2-client.mirror.111546.log.gz 2023-08-26T18:17:54.523 INFO:teuthology.orchestra.run.smithi124.stderr: 93.7% -- replaced with /var/log/ceph/cluster2-client.mirror.119010.log.gz 2023-08-26T18:17:56.939 INFO:teuthology.orchestra.run.smithi031.stderr: 94.4% -- replaced with /var/log/ceph/cluster1-osd.1.log.gz 2023-08-26T18:17:57.585 INFO:teuthology.orchestra.run.smithi031.stderr: 94.4% -- replaced with /var/log/ceph/cluster1-osd.0.log.gz 2023-08-26T18:17:58.688 INFO:teuthology.orchestra.run.smithi031.stderr: 94.5% -- replaced with /var/log/ceph/cluster1-osd.2.log.gz 2023-08-26T18:17:58.690 INFO:teuthology.orchestra.run.smithi031.stderr: 2023-08-26T18:17:58.690 INFO:teuthology.orchestra.run.smithi031.stderr:real 0m9.851s 2023-08-26T18:17:58.690 INFO:teuthology.orchestra.run.smithi031.stderr:user 0m27.428s 2023-08-26T18:17:58.690 INFO:teuthology.orchestra.run.smithi031.stderr:sys 0m0.964s 2023-08-26T18:18:35.704 INFO:teuthology.orchestra.run.smithi124.stderr: 93.7% -- replaced with /var/log/ceph/cluster2-osd.2.log.gz 2023-08-26T18:18:37.970 INFO:teuthology.orchestra.run.smithi124.stderr: 93.6% -- replaced with /var/log/ceph/cluster2-osd.1.log.gz 2023-08-26T18:18:41.456 INFO:teuthology.orchestra.run.smithi124.stderr: 93.5% -- replaced with /var/log/ceph/cluster2-osd.0.log.gz 2023-08-26T18:18:41.457 INFO:teuthology.orchestra.run.smithi124.stderr: 2023-08-26T18:18:41.457 INFO:teuthology.orchestra.run.smithi124.stderr:real 0m52.563s 2023-08-26T18:18:41.457 INFO:teuthology.orchestra.run.smithi124.stderr:user 2m46.043s 2023-08-26T18:18:41.458 INFO:teuthology.orchestra.run.smithi124.stderr:sys 0m6.972s 2023-08-26T18:18:41.459 INFO:tasks.ceph:Archiving logs... 2023-08-26T18:18:41.459 DEBUG:teuthology.misc:Transferring archived files from smithi031:/var/log/ceph to /home/teuthworker/archive/yuriw-2023-08-25_17:40:36-rbd-main-distro-default-smithi/7380431/remote/smithi031/log 2023-08-26T18:18:41.461 DEBUG:teuthology.orchestra.run.smithi031:> sudo tar c -f - -C /var/log/ceph -- . 2023-08-26T18:18:45.085 DEBUG:teuthology.misc:Transferring archived files from smithi124:/var/log/ceph to /home/teuthworker/archive/yuriw-2023-08-25_17:40:36-rbd-main-distro-default-smithi/7380431/remote/smithi124/log 2023-08-26T18:18:45.086 DEBUG:teuthology.orchestra.run.smithi124:> sudo tar c -f - -C /var/log/ceph -- . 2023-08-26T18:19:09.188 ERROR:teuthology.run_tasks:Manager failed: ceph Traceback (most recent call last): File "/home/teuthworker/src/github.com_ceph_ceph-c_e7eb9853fddaf661c68f207346131a7e00d8ed2f/qa/tasks/ceph.py", line 1940, in task yield File "/home/teuthworker/src/git.ceph.com_teuthology_449a1bc2027504e7b3c3d7b30fa4178906581da7/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ next(self.gen) File "/home/teuthworker/src/github.com_ceph_ceph-c_e7eb9853fddaf661c68f207346131a7e00d8ed2f/qa/tasks/rbd_mirror_thrash.py", line 216, in task raise RuntimeError('error during thrashing') RuntimeError: error during thrashing During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_449a1bc2027504e7b3c3d7b30fa4178906581da7/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/github.com_ceph_ceph-c_e7eb9853fddaf661c68f207346131a7e00d8ed2f/qa/tasks/ceph.py", line 1950, in task ctx.managers[config['cluster']].wait_for_clean() File "/home/teuthworker/src/github.com_ceph_ceph-c_e7eb9853fddaf661c68f207346131a7e00d8ed2f/qa/tasks/ceph_manager.py", line 2742, in wait_for_clean num_active_clean = self.get_num_active_clean() File "/home/teuthworker/src/github.com_ceph_ceph-c_e7eb9853fddaf661c68f207346131a7e00d8ed2f/qa/tasks/ceph_manager.py", line 2574, in get_num_active_clean pgs = self.get_pg_stats() File "/home/teuthworker/src/github.com_ceph_ceph-c_e7eb9853fddaf661c68f207346131a7e00d8ed2f/qa/tasks/ceph_manager.py", line 2340, in get_pg_stats out = self.raw_cluster_cmd('pg', 'dump', '--format=json') File "/home/teuthworker/src/github.com_ceph_ceph-c_e7eb9853fddaf661c68f207346131a7e00d8ed2f/qa/tasks/ceph_manager.py", line 1595, in raw_cluster_cmd return self.run_cluster_cmd(**kwargs).stdout.getvalue() File "/home/teuthworker/src/github.com_ceph_ceph-c_e7eb9853fddaf661c68f207346131a7e00d8ed2f/qa/tasks/ceph_manager.py", line 1586, in run_cluster_cmd return self.controller.run(**kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_449a1bc2027504e7b3c3d7b30fa4178906581da7/teuthology/orchestra/remote.py", line 522, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_449a1bc2027504e7b3c3d7b30fa4178906581da7/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_449a1bc2027504e7b3c3d7b30fa4178906581da7/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_449a1bc2027504e7b3c3d7b30fa4178906581da7/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi031 with status 124: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 pg dump --format=json' 2023-08-26T18:19:09.195 DEBUG:teuthology.run_tasks:Unwinding manager install 2023-08-26T18:19:09.210 ERROR:teuthology.contextutil:Saw exception from nested tasks Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_449a1bc2027504e7b3c3d7b30fa4178906581da7/teuthology/contextutil.py", line 32, in nested yield vars File "/home/teuthworker/src/git.ceph.com_teuthology_449a1bc2027504e7b3c3d7b30fa4178906581da7/teuthology/task/install/__init__.py", line 626, in task yield File "/home/teuthworker/src/git.ceph.com_teuthology_449a1bc2027504e7b3c3d7b30fa4178906581da7/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ next(self.gen) File "/home/teuthworker/src/github.com_ceph_ceph-c_e7eb9853fddaf661c68f207346131a7e00d8ed2f/qa/tasks/rbd_mirror_thrash.py", line 216, in task raise RuntimeError('error during thrashing') RuntimeError: error during thrashing 2023-08-26T18:19:09.218 INFO:teuthology.task.install.util:Removing shipped files: /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer... 2023-08-26T18:19:09.218 DEBUG:teuthology.orchestra.run.smithi031:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2023-08-26T18:19:09.240 DEBUG:teuthology.orchestra.run.smithi124:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2023-08-26T18:19:09.332 DEBUG:teuthology.run_tasks:Unwinding manager clock 2023-08-26T18:19:09.344 INFO:teuthology.task.clock:Checking final clock skew... 2023-08-26T18:19:09.345 DEBUG:teuthology.orchestra.run.smithi031:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2023-08-26T18:19:09.348 DEBUG:teuthology.orchestra.run.smithi124:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2023-08-26T18:19:09.365 INFO:teuthology.orchestra.run.smithi031.stderr:bash: ntpq: command not found 2023-08-26T18:19:09.366 INFO:teuthology.orchestra.run.smithi124.stderr:bash: ntpq: command not found 2023-08-26T18:19:09.369 INFO:teuthology.orchestra.run.smithi031.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2023-08-26T18:19:09.370 INFO:teuthology.orchestra.run.smithi031.stdout:=============================================================================== 2023-08-26T18:19:09.370 INFO:teuthology.orchestra.run.smithi031.stdout:^+ hv01.front.sepia.ceph.com 2 8 377 224 +516us[ +527us] +/- 72ms 2023-08-26T18:19:09.370 INFO:teuthology.orchestra.run.smithi031.stdout:^* hv02.front.sepia.ceph.com 3 8 377 174 -610us[ -597us] +/- 65ms 2023-08-26T18:19:09.371 INFO:teuthology.orchestra.run.smithi031.stdout:^+ hv03.front.sepia.ceph.com 3 7 377 38 +380us[ +380us] +/- 60ms 2023-08-26T18:19:09.371 INFO:teuthology.orchestra.run.smithi031.stdout:^? hv04.front.sepia.ceph.com 0 10 0 - +0ns[ +0ns] +/- 0ns 2023-08-26T18:19:09.373 INFO:teuthology.orchestra.run.smithi124.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2023-08-26T18:19:09.373 INFO:teuthology.orchestra.run.smithi124.stdout:=============================================================================== 2023-08-26T18:19:09.373 INFO:teuthology.orchestra.run.smithi124.stdout:^+ hv01.front.sepia.ceph.com 2 8 377 226 +542us[ +567us] +/- 72ms 2023-08-26T18:19:09.374 INFO:teuthology.orchestra.run.smithi124.stdout:^* hv02.front.sepia.ceph.com 3 7 377 90 -585us[ -576us] +/- 66ms 2023-08-26T18:19:09.374 INFO:teuthology.orchestra.run.smithi124.stdout:^+ hv03.front.sepia.ceph.com 3 7 377 28 +395us[ +395us] +/- 60ms 2023-08-26T18:19:09.374 INFO:teuthology.orchestra.run.smithi124.stdout:^? hv04.front.sepia.ceph.com 0 10 0 - +0ns[ +0ns] +/- 0ns 2023-08-26T18:19:09.375 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2023-08-26T18:19:09.387 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2023-08-26T18:19:09.387 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2023-08-26T18:19:09.404 DEBUG:teuthology.orchestra.run.smithi031:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2023-08-26T18:19:09.514 DEBUG:teuthology.orchestra.run.smithi124:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2023-08-26T18:19:09.607 DEBUG:teuthology.orchestra.run.smithi031:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2023-08-26T18:19:09.642 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-08-26T18:19:09.643 DEBUG:teuthology.orchestra.run.smithi124:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2023-08-26T18:19:09.674 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-08-26T18:19:09.675 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2023-08-26T18:19:09.687 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2023-08-26T18:19:09.700 INFO:teuthology.task.internal:Duration was 18252.683501 seconds 2023-08-26T18:19:09.701 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2023-08-26T18:19:09.716 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2023-08-26T18:19:09.717 DEBUG:teuthology.orchestra.run.smithi031:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2023-08-26T18:19:09.720 DEBUG:teuthology.orchestra.run.smithi124:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2023-08-26T18:19:09.766 INFO:teuthology.orchestra.run.smithi124.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2023-08-26T18:19:09.768 INFO:teuthology.orchestra.run.smithi031.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2023-08-26T18:19:10.138 INFO:teuthology.task.internal.syslog:Checking logs for errors... 2023-08-26T18:19:10.139 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi031.front.sepia.ceph.com 2023-08-26T18:19:10.139 DEBUG:teuthology.orchestra.run.smithi031:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2023-08-26T18:19:10.165 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi124.front.sepia.ceph.com 2023-08-26T18:19:10.165 DEBUG:teuthology.orchestra.run.smithi124:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2023-08-26T18:19:10.187 INFO:teuthology.task.internal.syslog:Compressing syslogs... 2023-08-26T18:19:10.187 DEBUG:teuthology.orchestra.run.smithi031:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2023-08-26T18:19:10.208 DEBUG:teuthology.orchestra.run.smithi124:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2023-08-26T18:19:10.269 INFO:teuthology.task.internal.syslog:Gathering journactl -b0... 2023-08-26T18:19:10.270 DEBUG:teuthology.orchestra.run.smithi031:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2023-08-26T18:19:10.273 DEBUG:teuthology.orchestra.run.smithi124:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2023-08-26T18:19:10.385 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2023-08-26T18:19:10.403 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2023-08-26T18:19:10.404 DEBUG:teuthology.orchestra.run.smithi031:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2023-08-26T18:19:10.441 DEBUG:teuthology.orchestra.run.smithi124:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2023-08-26T18:19:10.478 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2023-08-26T18:19:10.499 DEBUG:teuthology.orchestra.run.smithi031:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2023-08-26T18:19:10.502 DEBUG:teuthology.orchestra.run.smithi124:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2023-08-26T18:19:10.525 INFO:teuthology.orchestra.run.smithi031.stdout:kernel.core_pattern = core 2023-08-26T18:19:10.547 INFO:teuthology.orchestra.run.smithi124.stdout:kernel.core_pattern = core 2023-08-26T18:19:10.564 DEBUG:teuthology.orchestra.run.smithi031:> test -e /home/ubuntu/cephtest/archive/coredump 2023-08-26T18:19:10.598 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-08-26T18:19:10.599 DEBUG:teuthology.orchestra.run.smithi124:> test -e /home/ubuntu/cephtest/archive/coredump 2023-08-26T18:19:10.620 WARNING:teuthology.task.internal:Found coredumps on ubuntu@smithi124.front.sepia.ceph.com, flagging run as failed 2023-08-26T18:19:10.620 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2023-08-26T18:19:10.634 INFO:teuthology.task.internal:Transferring archived files... 2023-08-26T18:19:10.636 DEBUG:teuthology.misc:Transferring archived files from smithi031:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/yuriw-2023-08-25_17:40:36-rbd-main-distro-default-smithi/7380431/remote/smithi031 2023-08-26T18:19:10.636 DEBUG:teuthology.orchestra.run.smithi031:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2023-08-26T18:19:10.682 DEBUG:teuthology.misc:Transferring archived files from smithi124:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/yuriw-2023-08-25_17:40:36-rbd-main-distro-default-smithi/7380431/remote/smithi124 2023-08-26T18:19:10.682 DEBUG:teuthology.orchestra.run.smithi124:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2023-08-26T18:19:47.195 INFO:teuthology.task.internal:Transferring binaries for coredumps... 2023-08-26T18:19:47.656 INFO:teuthology.task.internal: core looks like: /home/teuthworker/archive/yuriw-2023-08-25_17:40:36-rbd-main-distro-default-smithi/7380431/remote/smithi124/coredump/1693059429.130432.core.gz: gzip compressed data, was "1693059429.130432.core", last modified: Sat Aug 26 18:19:10 2023, max compression, original size modulo 2^32 543391744 2023-08-26T18:19:47.657 INFO:teuthology.task.internal:core is compressed, try accessing gzip file ... 2023-08-26T18:19:49.796 INFO:teuthology.task.internal: core looks like: /tmp/tmpf_of02_9: ELF 64-bit LSB core file, x86-64, version 1 (SYSV), SVR4-style, from 'rbd-mirror --foreground --cluster cluster1 --id mirror.2', real uid: 1000, effective uid: 1000, real gid: 1265, effective gid: 1265, execfn: '/usr/bin/rbd-mirror', platform: 'x86_64' 2023-08-26T18:19:49.856 INFO:teuthology.task.internal: dump_program: rbd-mirror --foreground --cluster cluster1 --id mirror.2 2023-08-26T18:19:49.857 DEBUG:teuthology.orchestra.run.smithi124:> which 'rbd-mirror --foreground --cluster cluster1 --id mirror.2' 2023-08-26T18:19:49.879 INFO:teuthology.orchestra.run.smithi124.stderr:/usr/bin/which: unrecognized option '--foreground' 2023-08-26T18:19:49.879 INFO:teuthology.orchestra.run.smithi124.stderr:/usr/bin/which: unrecognized option '--cluster' 2023-08-26T18:19:49.880 INFO:teuthology.orchestra.run.smithi124.stderr:/usr/bin/which: unrecognized option '--id' 2023-08-26T18:19:49.880 INFO:teuthology.orchestra.run.smithi124.stderr:/usr/bin/which: no cluster1 in (/home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin) 2023-08-26T18:19:49.880 INFO:teuthology.orchestra.run.smithi124.stdout:/usr/bin/rbd-mirror 2023-08-26T18:19:49.881 DEBUG:teuthology.orchestra.run:got remote process result: 2 2023-08-26T18:19:49.882 INFO:teuthology.orchestra.run.smithi124.stderr:/usr/bin/which: no mirror.2 in (/home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin) 2023-08-26T18:19:49.882 ERROR:teuthology.run_tasks:Manager failed: internal.archive Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_449a1bc2027504e7b3c3d7b30fa4178906581da7/teuthology/task/internal/__init__.py", line 392, in archive yield File "/home/teuthworker/src/git.ceph.com_teuthology_449a1bc2027504e7b3c3d7b30fa4178906581da7/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ next(self.gen) File "/home/teuthworker/src/github.com_ceph_ceph-c_e7eb9853fddaf661c68f207346131a7e00d8ed2f/qa/tasks/rbd_mirror_thrash.py", line 216, in task raise RuntimeError('error during thrashing') RuntimeError: error during thrashing During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_449a1bc2027504e7b3c3d7b30fa4178906581da7/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/git.ceph.com_teuthology_449a1bc2027504e7b3c3d7b30fa4178906581da7/teuthology/task/internal/__init__.py", line 420, in archive fetch_binaries_for_coredumps(path, rem) File "/home/teuthworker/src/git.ceph.com_teuthology_449a1bc2027504e7b3c3d7b30fa4178906581da7/teuthology/task/internal/__init__.py", line 345, in fetch_binaries_for_coredumps remote_path = remote.sh(['which', dump_program]).rstrip() File "/home/teuthworker/src/git.ceph.com_teuthology_449a1bc2027504e7b3c3d7b30fa4178906581da7/teuthology/orchestra/remote.py", line 96, in sh proc = self.run(**kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_449a1bc2027504e7b3c3d7b30fa4178906581da7/teuthology/orchestra/remote.py", line 522, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_449a1bc2027504e7b3c3d7b30fa4178906581da7/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_449a1bc2027504e7b3c3d7b30fa4178906581da7/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_449a1bc2027504e7b3c3d7b30fa4178906581da7/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi124 with status 2: "which 'rbd-mirror --foreground --cluster cluster1 --id mirror.2'" 2023-08-26T18:19:49.895 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2023-08-26T18:19:49.910 INFO:teuthology.task.internal:Not uploading archives. 2023-08-26T18:19:49.910 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2023-08-26T18:19:49.924 INFO:teuthology.task.internal:Tidying up after the test... 2023-08-26T18:19:49.924 DEBUG:teuthology.orchestra.run.smithi031:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2023-08-26T18:19:49.931 DEBUG:teuthology.orchestra.run.smithi124:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2023-08-26T18:19:49.950 INFO:teuthology.orchestra.run.smithi031.stdout: 262307 4 drwxr-xr-x 3 ubuntu ubuntu 4096 Aug 26 18:19 /home/ubuntu/cephtest 2023-08-26T18:19:49.951 INFO:teuthology.orchestra.run.smithi031.stdout: 262321 4 drwxr-xr-x 4 ubuntu ubuntu 4096 Aug 26 18:19 /home/ubuntu/cephtest/archive 2023-08-26T18:19:49.951 INFO:teuthology.orchestra.run.smithi031.stdout: 659006 4 drwxrwxr-x 2 ubuntu ubuntu 4096 Aug 26 18:19 /home/ubuntu/cephtest/archive/audit 2023-08-26T18:19:49.952 INFO:teuthology.orchestra.run.smithi031.stdout: 659285 240 -rw------- 1 ubuntu ubuntu 245100 Aug 26 18:19 /home/ubuntu/cephtest/archive/audit/audit.log.gz 2023-08-26T18:19:49.952 INFO:teuthology.orchestra.run.smithi031.stdout: 262323 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Aug 26 18:19 /home/ubuntu/cephtest/archive/syslog 2023-08-26T18:19:49.952 INFO:teuthology.orchestra.run.smithi031.stdout: 262157 4 -rw-rw-rw- 1 ubuntu ubuntu 611 Aug 26 18:17 /home/ubuntu/cephtest/archive/syslog/kern.log.gz 2023-08-26T18:19:49.952 INFO:teuthology.orchestra.run.smithi031.stdout: 262200 124 -rw-rw-rw- 1 ubuntu ubuntu 124483 Aug 26 18:19 /home/ubuntu/cephtest/archive/syslog/misc.log.gz 2023-08-26T18:19:49.953 INFO:teuthology.orchestra.run.smithi031.stdout: 262209 120 -rw-rw-r-- 1 ubuntu ubuntu 121149 Aug 26 18:19 /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2023-08-26T18:19:49.954 INFO:teuthology.orchestra.run.smithi124.stdout: 262320 4 drwxr-xr-x 4 ubuntu ubuntu 4096 Aug 26 18:19 /home/ubuntu/cephtest 2023-08-26T18:19:49.954 INFO:teuthology.orchestra.run.smithi124.stdout: 262325 4 drwxr-xr-x 5 ubuntu ubuntu 4096 Aug 26 18:19 /home/ubuntu/cephtest/archive 2023-08-26T18:19:49.954 INFO:teuthology.orchestra.run.smithi124.stdout: 659018 4 drwxrwxr-x 2 ubuntu ubuntu 4096 Aug 26 18:19 /home/ubuntu/cephtest/archive/audit 2023-08-26T18:19:49.955 INFO:teuthology.orchestra.run.smithi124.stdout: 659022 244 -rw------- 1 ubuntu ubuntu 247808 Aug 26 18:19 /home/ubuntu/cephtest/archive/audit/audit.log.gz 2023-08-26T18:19:49.955 INFO:teuthology.orchestra.run.smithi124.stdout: 262326 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Aug 26 14:17 /home/ubuntu/cephtest/archive/coredump 2023-08-26T18:19:49.955 INFO:teuthology.orchestra.run.smithi124.stdout: 262257 52200 -rw------- 1 ubuntu ubuntu 543391744 Aug 26 14:17 /home/ubuntu/cephtest/archive/coredump/1693059429.130432.core 2023-08-26T18:19:49.955 INFO:teuthology.orchestra.run.smithi124.stdout: 262215 52724 -rw------- 1 ubuntu ubuntu 642310144 Aug 26 14:17 /home/ubuntu/cephtest/archive/coredump/1693059429.131574.core 2023-08-26T18:19:49.956 INFO:teuthology.orchestra.run.smithi124.stdout: 262327 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Aug 26 18:19 /home/ubuntu/cephtest/archive/syslog 2023-08-26T18:19:49.956 INFO:teuthology.orchestra.run.smithi124.stdout: 262157 4 -rw-rw-rw- 1 ubuntu ubuntu 814 Aug 26 18:15 /home/ubuntu/cephtest/archive/syslog/kern.log.gz 2023-08-26T18:19:49.956 INFO:teuthology.orchestra.run.smithi124.stdout: 262200 116 -rw-rw-rw- 1 ubuntu ubuntu 118542 Aug 26 18:19 /home/ubuntu/cephtest/archive/syslog/misc.log.gz 2023-08-26T18:19:49.957 INFO:teuthology.orchestra.run.smithi124.stdout: 262209 112 -rw-rw-r-- 1 ubuntu ubuntu 114159 Aug 26 18:19 /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2023-08-26T18:19:49.957 INFO:teuthology.orchestra.run.smithi124.stdout: 659020 4 drwxrwxr-x 3 ubuntu ubuntu 4096 Aug 26 13:32 /home/ubuntu/cephtest/mnt.cluster1.mirror 2023-08-26T18:19:49.957 INFO:teuthology.orchestra.run.smithi124.stdout: 659021 4 drwxrwxr-x 3 ubuntu ubuntu 4096 Aug 26 13:32 /home/ubuntu/cephtest/mnt.cluster1.mirror/client.mirror 2023-08-26T18:19:49.957 INFO:teuthology.orchestra.run.smithi124.stdout: 1052551 4 drwxrwxr-x 2 ubuntu ubuntu 4096 Aug 26 13:32 /home/ubuntu/cephtest/mnt.cluster1.mirror/client.mirror/tmp 2023-08-26T18:19:49.958 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-08-26T18:19:49.959 INFO:teuthology.orchestra.run.smithi031.stderr:rmdir: failed to remove '/home/ubuntu/cephtest': Directory not empty 2023-08-26T18:19:49.960 ERROR:teuthology.run_tasks:Manager failed: internal.base Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_449a1bc2027504e7b3c3d7b30fa4178906581da7/teuthology/task/internal/__init__.py", line 48, in base yield File "/home/teuthworker/src/git.ceph.com_teuthology_449a1bc2027504e7b3c3d7b30fa4178906581da7/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ next(self.gen) File "/home/teuthworker/src/github.com_ceph_ceph-c_e7eb9853fddaf661c68f207346131a7e00d8ed2f/qa/tasks/rbd_mirror_thrash.py", line 216, in task raise RuntimeError('error during thrashing') RuntimeError: error during thrashing During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_449a1bc2027504e7b3c3d7b30fa4178906581da7/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/git.ceph.com_teuthology_449a1bc2027504e7b3c3d7b30fa4178906581da7/teuthology/task/internal/__init__.py", line 53, in base run.wait( File "/home/teuthworker/src/git.ceph.com_teuthology_449a1bc2027504e7b3c3d7b30fa4178906581da7/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_449a1bc2027504e7b3c3d7b30fa4178906581da7/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_449a1bc2027504e7b3c3d7b30fa4178906581da7/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi031 with status 1: 'find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest' 2023-08-26T18:19:49.970 DEBUG:teuthology.run_tasks:Unwinding manager kernel 2023-08-26T18:19:49.984 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2023-08-26T18:19:49.999 DEBUG:teuthology.run_tasks:Exception was not quenched, exiting: RuntimeError: error during thrashing 2023-08-26T18:19:49.999 INFO:teuthology.nuke:Checking targets against current locks 2023-08-26T18:19:50.001 INFO:teuthology.orchestra.run.smithi124.stderr:rmdir: failed to remove '/home/ubuntu/cephtest': Directory not empty 2023-08-26T18:19:50.029 DEBUG:teuthology.nuke:shortname: smithi031 2023-08-26T18:19:50.029 INFO:teuthology.task.internal.check_lock:Checking locks... 2023-08-26T18:19:50.057 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi031.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yuriw-2023-08-25_17:40:36-rbd-main-distro-default-smithi/7380431', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'rhel', 'os_version': '8.6', 'arch': 'x86_64', 'locked': True, 'locked_since': '2023-08-26 13:09:35.383564', 'locked_by': 'scheduled_yuriw@teuthology', 'mac_address': None, 'ssh_pub_key': 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCpUxo5ay/ucdkX9rleNMm8Ofp24XRXUtJPOK3Ji3XBpTiUgmmwneaY6316+3CG+k4K0p3IebanuyTvZApjm02/dNCvu8yD3rjEpWVgIuqYH8O3lEdN9DXcCX+7xHJE0m18+ZPsJQVsNrNJxY+5zqyNnBc8jpZq2LStsWwPOKPWhauANRkOBFSGDoLppWZbLx/yJs+R0qoNLLK9LpKQTUnqsapXvbD27OpPahzj/qbNxFPPyY6XrrldZ6qEO+EjJOujFLH3gzv0kJu10DCuzxrpz6ston4RPi1O3GgcHnF4Mqr/W9VwZ94UTMh+k4dIFRVqJEKrcwRHa5kFsvH/WhVx0Ygr/MiV/Q86mmV7JT+/MpzIRTtWyei5L9IBeZIE9cwzj+mOk6SD/t1roIVEGVV/uF2AyFDVBPr6F8oZ822OGpUyyaDsWTitPNdTX4TC3zdSrLc9tyDa1P74eH/ZLeTSf+C6pM/cepOhQaa6XzpDiPSuJUm9HOQxUlEAvcqa7Ac='} 2023-08-26T18:19:50.061 DEBUG:teuthology.nuke:shortname: smithi124 2023-08-26T18:19:50.062 INFO:teuthology.task.internal.check_lock:Checking locks... 2023-08-26T18:19:50.086 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi124.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yuriw-2023-08-25_17:40:36-rbd-main-distro-default-smithi/7380431', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'rhel', 'os_version': '8.6', 'arch': 'x86_64', 'locked': True, 'locked_since': '2023-08-26 13:09:35.381920', 'locked_by': 'scheduled_yuriw@teuthology', 'mac_address': None, 'ssh_pub_key': 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCpUxo5ay/ucdkX9rleNMm8Ofp24XRXUtJPOK3Ji3XBpTiUgmmwneaY6316+3CG+k4K0p3IebanuyTvZApjm02/dNCvu8yD3rjEpWVgIuqYH8O3lEdN9DXcCX+7xHJE0m18+ZPsJQVsNrNJxY+5zqyNnBc8jpZq2LStsWwPOKPWhauANRkOBFSGDoLppWZbLx/yJs+R0qoNLLK9LpKQTUnqsapXvbD27OpPahzj/qbNxFPPyY6XrrldZ6qEO+EjJOujFLH3gzv0kJu10DCuzxrpz6ston4RPi1O3GgcHnF4Mqr/W9VwZ94UTMh+k4dIFRVqJEKrcwRHa5kFsvH/WhVx0Ygr/MiV/Q86mmV7JT+/MpzIRTtWyei5L9IBeZIE9cwzj+mOk6SD/t1roIVEGVV/uF2AyFDVBPr6F8oZ822OGpUyyaDsWTitPNdTX4TC3zdSrLc9tyDa1P74eH/ZLeTSf+C6pM/cepOhQaa6XzpDiPSuJUm9HOQxUlEAvcqa7Ac='} 2023-08-26T18:19:50.116 INFO:teuthology.orchestra.console.smithi031:Power off 2023-08-26T18:19:50.117 DEBUG:teuthology.orchestra.console.smithi031:pexpect command: ipmitool -H smithi031.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2023-08-26T18:19:50.140 DEBUG:teuthology.orchestra.console.smithi031:power off output: Chassis Power Control: Down/Off 2023-08-26T18:19:50.140 DEBUG:teuthology.orchestra.console.smithi031:pexpect command: ipmitool -H smithi031.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-08-26T18:19:50.157 INFO:teuthology.orchestra.console.smithi124:Power off 2023-08-26T18:19:50.158 DEBUG:teuthology.orchestra.console.smithi124:pexpect command: ipmitool -H smithi124.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2023-08-26T18:19:50.168 DEBUG:teuthology.orchestra.console.smithi031:check power output: Chassis Power is on 2023-08-26T18:19:50.181 DEBUG:teuthology.orchestra.console.smithi124:power off output: Chassis Power Control: Down/Off 2023-08-26T18:19:50.181 DEBUG:teuthology.orchestra.console.smithi124:pexpect command: ipmitool -H smithi124.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-08-26T18:19:50.208 DEBUG:teuthology.orchestra.console.smithi124:check power output: Chassis Power is on 2023-08-26T18:19:54.170 DEBUG:teuthology.orchestra.console.smithi031:pexpect command: ipmitool -H smithi031.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-08-26T18:19:54.209 DEBUG:teuthology.orchestra.console.smithi124:pexpect command: ipmitool -H smithi124.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-08-26T18:19:54.282 DEBUG:teuthology.orchestra.console.smithi031:check power output: Chassis Power is on 2023-08-26T18:19:54.320 DEBUG:teuthology.orchestra.console.smithi124:check power output: Chassis Power is on 2023-08-26T18:19:58.284 DEBUG:teuthology.orchestra.console.smithi031:pexpect command: ipmitool -H smithi031.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-08-26T18:19:58.321 DEBUG:teuthology.orchestra.console.smithi124:pexpect command: ipmitool -H smithi124.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-08-26T18:19:58.394 DEBUG:teuthology.orchestra.console.smithi031:check power output: Chassis Power is on 2023-08-26T18:19:58.432 DEBUG:teuthology.orchestra.console.smithi124:check power output: Chassis Power is off 2023-08-26T18:19:58.533 INFO:teuthology.orchestra.console.smithi124:Power off completed 2023-08-26T18:20:02.400 DEBUG:teuthology.orchestra.console.smithi031:pexpect command: ipmitool -H smithi031.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-08-26T18:20:02.509 DEBUG:teuthology.orchestra.console.smithi031:check power output: Chassis Power is off 2023-08-26T18:20:02.610 INFO:teuthology.orchestra.console.smithi031:Power off completed 2023-08-26T18:20:02.717 INFO:teuthology.run:Summary data: description: rbd/mirror-thrash/{base/install clients/mirror cluster/{2-node openstack} msgr-failures/few objectstore/bluestore-stupid policy/simple rbd-mirror/four-per-cluster supported-random-distro$/{rhel_8} workloads/rbd-mirror-snapshot-stress-workunit-exclusive-lock} duration: 18252.683500528336 failure_reason: 'Command failed (workunit test rbd/rbd_mirror_stress.sh) on smithi124 with status 1: ''mkdir -p -- /home/ubuntu/cephtest/mnt.cluster1.mirror/client.mirror/tmp && cd -- /home/ubuntu/cephtest/mnt.cluster1.mirror/client.mirror/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=e7eb9853fddaf661c68f207346131a7e00d8ed2f TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster cluster1" CEPH_ID="mirror" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.cluster1.client.mirror CEPH_ROOT=/home/ubuntu/cephtest/clone.cluster1.client.mirror CEPH_MNT=/home/ubuntu/cephtest/mnt.cluster1.mirror CEPH_ARGS=\''\'' MIRROR_IMAGE_MODE=snapshot MIRROR_POOL_MODE=image RBD_IMAGE_FEATURES=layering,exclusive-lock RBD_MIRROR_INSTANCES=4 RBD_MIRROR_USE_EXISTING_CLUSTER=1 RBD_MIRROR_USE_RBD_MIRROR=1 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 6h /home/ubuntu/cephtest/clone.cluster1.client.mirror/qa/workunits/rbd/rbd_mirror_stress.sh''' flavor: default owner: scheduled_yuriw@teuthology sentry_event: https://sentry.ceph.com/organizations/ceph/?query=6f5588fc58b2415f80c823da166634a5 status: fail success: false 2023-08-26T18:20:02.718 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2023-08-26T18:20:02.795 INFO:teuthology.run:FAIL